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This  Handbook  has  been  written 


WHAT 

TECHNIQUES 

ARE 

AVAILABLE 


for  those  program  management  personnel  who 
have  an  interest,  but  little  (if  any)  prior 
experience  in  accomplishing  a  quantitative 
risk  assessment  of  an  acquisition  program. 


HOW  DO 
I  SELECT  A 
TECHNIQUE? 


Thus,  the  Handbook  is  organized  to  provide  the  "layman" 
with  an  overview  of  risk  assessment  as  applied  to  defense 
acquisition  programs  in  the  first  six  chapters.  Support¬ 
ing  details  are  provided  in  the  Appendices. 
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'The  primary  objectives  of  this  handbook  are  to  make  the 
reader  aware  of  the  risk  assessment  techniques  being  used  by 
Department  of  Defense  organizations,  to  alert  the  reader  to  the 
advantages  and  disadvantages  of  these  techniques,  and  to  assist 
him'j^-in  applying  risk  assessment  to  his  acquisition  program. 

The  handbook  is  intended  to  be  a  practical  guide  and 
reference  for  program  management  personnel  -  not  a  textbook 
dealing  with  the  theories  supporting  risk  analysis,  nor  a  user's 
manual  for  applying  any  particular  techniques.  Thus,  the  hand¬ 
book  is  organized  to  address,  in  summary,  the  most  important 
questions  to  program  management  personnel,  i.e.,  Why  do  a  risk 
assessment?  What  techniques  are  available?  How  do  I  select  and 
implement  a  technique?  These  questions  are  answered  in  the  first 
six  chapters.  This  summary-level  material  is  supported  by  a 
series  of  Appendices  that  provide  detailed  discussions  of  the 
techniques  in  use,  the  service  regulations  pertaining  to  risk 
assessments,  a  glossary  of  terms,  and  a  structured  bibliography^ 

Much  of  the  basic  information  contained  in  this  handbook  was\ 
collected  during  a  survey  of  program  management  offices  and  other 
service  acquisition  activities.  The  time  that  individuals  in 
these  offices  allocated  from  their  busy  schedules  to  answer 
questions,  locate  related  documentation  and  provide  follow-up 
data  should  be  recognized  and  appreciated  by  those  who  make  use 
of  this  handbook.  The  Defense  Systems  Management  College  (DSMC) 
extends  a  "thank-you"  to  these  individuals. 

1/  Whenever  in  this  handbook  ~u'an,"  "men,"  or  their  related 
pronouns  appear  either  as  words  or  parts  of  words  (other  than 
with  obvious  reference  to  named  male  individuals),  they  have  been 
used  for  literary  purposes  and  are  meant  in  their  generic  sense. 
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CHAPTER  I 
INTRODUCTION 

A.  PURPOSE  AND  APPROACH 

Former  Deputy  Secretary  of  Defense  Frank  C.  Carlucci,  III, 
published  a  memorandum  shortly  after  taking  office  in  March  1981, 
with  the  objective  and  title  of  "Improving  the  Acquisition 
Process"  (reference  Cl]) . ^  This  memorandum  included  32 
"initiatives".  Initiative  Number  11  required  Department  of 
Defense  (DOD)  action  to  increase  the  visibility  of  technical  risk 
in  budgets  of  weapon  systems  acquisition  programs.  Specifically, 
the  memorandum  required  the  Services  (among  other  things)  to 
"Incorporate  the  Use  of  Budget  Funds  for  Technological  Risk",  and 
recommended  an  increase  in  "DOD  efforts  to  quantify  and  expand 
the  use  of  budgeted  funds  to  deal  with  uncertainty" .  This  hand¬ 
book,  therefore,  has  been  written  to  familiarize  program  manage¬ 
ment  personnel  with  the  concepts  and  techniques  of  quantitative 
risk  assessment,  the  conditions  under  which  different  techniques 
are  appropriate,  what  resources  are  required,  what  assistance  the 
different  techniques  provide,  and  what  administrative  procedures 
are  prescribed  for  accomplishing  quantitative  risk  assessment. 

This  handbook  is  intended  to  provide  program  managers2  with 
a  practical  guide  into  this  highly  technical  area,  so  that  they 
will  see  the  value  of  using  formal  quantitative  risk  assessment 
to  assist  them  in  internal  management  decision-making  and  in 
explaining  their  programs  to  higher  review  levels. 

The  methods  used  in  this  handbook  to  accomplish  its  purpose 
are  first,  to  present  a  simplified  explanation  of  the  theory 
behind  formal  quantitative  risk  assessment;  second,  to 


■^Bracketed  numbers  in  the  text  indicate  references  listed  at 
the  end  of  respective  chapters. 

2 

For  brevity  the  term  "program  manager"  will  be  used  for 
either  "program",  "project",  or  "product  manager",  or  for 
"program  coordinator",  "project  engineer"  or  other  variants. 

1-1 


ca 


1 


demonstrate  to  program  management  personnel  that  they  will  find 
quantitative  risk  assessment  advantageous  to  use;  third ,  to 
describe  individual  techniques  at  a  level  of  detail  permitting 
appreciation  and  intelligent  selection  (but  not  necessarily  per¬ 
sonal  utilization);  and  last,  to  inform  program  mnagement  person¬ 
nel  of  considerations  relating  to  obtaining  the  support  they  may 
need  to  undertake  an  assessment. 

B.  SCOPE 

Risk  analysis  has  been  defined  (e.g.,  reference  [2])  as  a 
concept  naving  a  broader  definition  than  that  of  risk  assess¬ 
ment,  3  encompassing  risk  assessment,  risk  reductions,  and  risk 
management,  as  shown  in  the  right  side  of  Figure  1.  The  figure 
shows  one  way  of  viewing  the  role  of  risk  assessment  in  program 
management . 

This  handbook  addresses  formai,  quantitative  assessment  of 
program  risk,3 4  generally  leaving  the  creation  and  selection  of 
risk-reducing  program  alternatives  and  the  implementation  and 
control  of  the  selected  alternative ( s )  to  other  guidance  sources. 

Since  the  motivation  for  this  handbook  is  Mr.  Carlucci's 
initiative  on  budgeting  for  technological  risk,  the  emphasis  will 
be  on  methods  which  assist  in  the  formation  of  budgets.  However, 
the  interdependence  of  schedules,  technological  developments,  and 
costs  necessitates  some  discussion  of  risk  assessment  methods 
concerned  with  all  three. 


3 Risk  assessment  will  be  described  in  Chapter  II. 

4 Note  that  safety,  or  risk  of  accident,  is  not  a  addressed  in 
this  handbook. 
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Figure  1 

RISK  ANALYSIS  COMPONENTS 

This  handbook  is  intended  to  be  a  reference  and  guide  for 
progiam  management  personnel.  If,  at  times,  liberties  are  taken 
with  mathematical  rigor,  it  is  for  the  purpose  of  clarifying  con¬ 
cepts  for  people  who  must  absorb  and  appreciate  them  enough  to 
decide  on  employing  them,  but  need  not  attain  the  analyst's 
requirement  for  rigor  or  for  understanding  of  subtle  issues  of 
theory.  The  reader  who  wishes  to  pursue  any  of  the  topics  intro¬ 
duced  herein  can  find  depth,  rigor,  and  interesting  exposition  in 
the  works  designated  as  "core"  (indicated  with  an  asterisk)  in 
the  bibliography,  Appendix  A. 

C.  SOURCES 

The  statements  made  in  this  handbook  regarding  practical  con¬ 
siderations  in  the  acquisition  environment  are  derived  from  an 
extensive  survey  of  program  management  and  analytical  support 
offices  in  the  military  services.  The  offices  were  selected  to 
represent  a  wide  range  of  technologies,  management  office 
"strengths",  technical  complexities,  costs,  and  maturities.  More 
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than  40  program  offices,  8  analytical  support  offices,  various 
headquarters  within  DOD,  and  academic  and  industry  researchers, 
constitute  the  sources  contributing  knowledge  of  the  techniques 
described. 
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CHAPTER  II 


WHAT  IS  RISK  ASSESSMENT? 


A.  GENERAL 

The  Introduction  (Chapter  I)  referred  to  "formal,  quan¬ 
titative  risk  assessment"  and  stated  that  it  was  not  identical  to 
risk  analysis.  The  questions  remain,  "What  is  risk  assessment, 
and  why  are  the  modifiers  'formal'  and  'quantitative'  used?" 
This  chapter  is  designed  to  answer  those  questions. 

B.  HOW  IS  IT  FORMAL? 


The  reader  may  be  familiar  with  assessments  in  which  the 
author  has  used  objectives  "high",  "medium",  or  "low"  to  express 
a  general  assessment  of  risk.  There  are  undoubtedly  circumstan¬ 
ces  under  which  such  non-quantitative  appraisals  are  appropriate, 
but  the  thrust  of  this  handbook  is  to  promote  the  language  of 
probability  and  the  use  of  its  associated  mathematics  in  risk 
analysis. 

All  program  managers  are  conscious  that  acquisition  programs 
typically  involve  a  risk  of  failure  to  meet  cost,  schedule,  or 
technical  performance  goals,  and  that  their  job  is  to  reduce  and 
control  that  risk.  The  risk  assessment  that  this  handbook 
describes  differs  from  the  top-level  intuitive  assessment 
exemplified  by  the  "high",  "medium",  "low,"  statement  in  that  it 
results  in  statements  of  risk  that  are  quantified  in  the  language 
of  mathematical  probability  and  that  it  commences  with  explicit 
examination  of  program  elements  at  some  specified  level  of  detail. 
By  examining  the  detailed  elements  of  the  program  it  is  possible 
to  find  experts  highly  familiar  with  each  element,  experts  who 
have  experience  in  the  field  of  which  the  program  element  is  an 
example. 


Most  of  the  current  systems  analysis  techniques  are  based  on 
the  concept  that  problems  having  large  numbers  of  elements  which 


interrelate  in  complex  manners  can  best  be  attacked  by  examining 
the  elements  in  detail,  determining  individual  relationships,  and 
formulating  a  model  from  those  building  blocks  to  determine  the 
behavior  of  the  model.  This  requires  formal  analysis.  The 
assumption  is  that  most  minds  are  less  able  to  comprehend  the 
whole  of  a  problem  than  to  comprehend  all  elements  individually. 
Determining  the  cost,  schedule  and  technical  risks  in  an  acquisi¬ 
tion  program  is  certainly  a  problem  involving  a  large  number  of 
elements  having  complex  interrelationships,  thus  the  adjective 
"formal"  applied  to  risk  assessment  signifies  that  the  analytic 
technique  will  be  used. 

C.  HOW  IS  IT  QUANTITATIVE? 

By  now  the  reader  should  see  that  this  handbook's  risk 
assessment  will  be  cased  on  examination  of  program  elements, 
using  experts,  in  order  to  synthesize  a  model  of  some  kind.  The 
next  characteristic  of  risk  assessment  to  be  addressed  is  the 
means  of  reducing  the  knowledge  of  the  program  elements  experts  to 
quantitative  expressions  of  uncertainty. 

The  need  for  qurntifying  experts'  inputs  arises  from  the 
complexity  of  interrelationships  between  program  elements.  If 
the  interrelationships  were  simple,  implications  might  be  easy 
to  see,  but  since  in  most  cases  they  are  complex,  it  is  advan¬ 
tageous  to  establish  a  mathematical  description  (model)  so  that 
magnitudes  of  the  influences  of  their  characteristics  and 
interrelationships  can  be  determined. 

1.  Subjective  Probability 

Before  discussing  mathematical  probability  methods  for  using 
experts’  knowledge,  it  will  be  helpful  to  clarify  what  is  meant 
by  "risk".  In  this  handbook  "risk"  is  the  term  used  to  denote 
the  probability  of  an  event  and  its  consequence.  1  When  an 


iln  many  disciplines  "risk"  and  uncertainty  are  given  distinct 
definitions,  and  program  decisions  under  those  definitions 
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event  has  a  high  probability  of  occurrence  and  an  unimportant 
consequence,  we  say  its  risk  is  low.  Similarly  when  its  con¬ 
sequence  is  quite  unpleasant  but  is  probability  of  occurrence  is 
very  low,  we  also  say  its  risk  is  low.  On  the  other  hand,  the 
combination  of  a  moderate  probability  of  occurrence  but  impor¬ 
tantly  undesirable  consequence  will  cause  the  risk  to  be  termed 
"high."  The  concept  of  risk  used  in  this  handbook  is  that  of  the 
probability  and  consequence  of  not  achieving  some  defined  program 
goal  (such  as  cost,  schedule,  or  technical  performance). 

Acquisition  programs  can  be  considered  "experiments"  that 
have  only  indirectly  related  precedents,  are  extremely  complex, 
and  will  be  carried  out  only  once.  A  mathematical  person  might 
say  that  this  "experiment"  is  fundamentally  different  from  the 
throwing  of  dice,  an  experiment  which  has  clearly  definable  pre¬ 
cedents,  and  therefore  although  mathematical  probability  is 
appropriate  to  dice  throwing,  it  is  not  appropriate  to  acquisi¬ 
tion  programs.  This  argument  is  based  on  one  viewpoint  of  the 
theory  of  probability.  Another  viewpoint  holds  that  probability 
is  a  statement  about  lack  of  knowledge,  and  that  when  a  person 
assigns  a  certain  probability  number  to  the  occurrence  of  an 
event,  he  is  saying  that  he  cannot  predict  the  event's  outcome, 
but  is  willing  to  take  some  action  (e.g.,  make  a  wager)  so  that 
his  payoff  and  hazard  are  related  by  this  probability  number. 
This  viewpoint  has  come  to  be  called  "subjective"  probability, 
and  although  not  all  reputable  statisticians  accept  the  legiti¬ 
macy  of  subjective  probability,  the  validity  of  the  theory  has 
received  a  great  amount  of  attention,  and  there  is  strong  support 
in  the  statistical  community  for  acceptance  of  the  concept. 
Thus,  the  application  of  probabilistic  assessments  to  the 
acquisition  process  has  a  strong  rational  basis.  The  interested 


are  decisions  under  uncertainty,  not  risk.  Although  the 
distinction  is  an  important  one,  it  is  less  confusing  to 
avoid  it  here. 
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reader  with  a  background  in  undergraduate  probability  theory  will 
find  a  clear  development  of  subjective  probability  in  [1], 

2.  Statements  of  Probabilit 


Most  readers  have  been  exposed  to  the  familiar  "bell-shaped" 
(or  "normal")  curve  (such  as  that  in  Figure  2)  used  for  such  pur¬ 
poses  as  describing  the  chance  that  a  bomb  will  fall  to  the  left 
or  right  of  a  target  by  a  given  distance  or  that  the  product  of  a 
machine  will  differ  from  its  specified  size  by  a  given  amount. 
The  kind  of  curve  shown  in  Figure  2  is  called  a  Probability 
Density  Function  (PDF)  and  is  just  one  of  the  curve  types 
described  in  this  handbook. 


Probability 


Distance  or  Size 


Figure  2 

PROBABILITY  DENSITY  FUNCTION  (PDF) 

The  other  type  is  the  Cumulative  Distribution  Function  (CDF) 
as  shown  in  Figure  3. 
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Distance  or  Six* 

Figure  3 

CUMULATIVE  DISTRIBUTION  FUNCTION  (CDF) 


Figures  2  and  3  are  the  types  of  curves  used  when  the  outcome 
of  an  experiment  (bomb  drop  or  part  manufacture)  can  be  a  number 
anywhere  within  a  range,  not  just  at  certain  values  (like  throws 
of  dice).  For  the  latter  type  of  experiment,  the  bar  graph 
(histogram)  like  Figure  4  is  used  instead  of  the  PDF. 


Probability 


Outcoma  of  Throw 

Figure  4 

PROBABILITY  MASS  FUNCTION  ( PMF ) 
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This  bar  graph  is  called  a  Probability  Mass  Function,  (PMF), 
however  since  most  of  the  ideas  in  this  handbook  can  be  con¬ 
ceived  in  terms  of  continuous  curves  (as  represented  by  Figures  2 
and  3),  Probability  Mass  Functions  will  not  be  much  discussed  in 
this  handbook. 
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Figure  5  is  another  example  of  a  symmetric  or  normal  PDF, 
representing,  for  example,  the  probability  of  final  program  cost. 


Figure  5 

SYMMETRIC  PDF  FOR  ELEMENT  COST 

In  a  PDF,  the  probability  of  an  event  occurring  within  a 
range  of  values  is  represented  by  the  area  under  the  curve 
within  the  range  of  interest.  Therefore,  since  the  two  shaded 
areas  in  Figure  5  are  equal,  the  curve  indicates  that  the  proba¬ 
bility  that  the  cost  will  be  between  4  and  5  million  is  the  same 
as  the  probability  that  it  will  be  between  7  and  8  million. 

In  many  of  the  concepts  discussed  in  this  handbook  neither 
normal  PDF's  or  other  symmetric  ones,  such  as  represented  by 
Figures  2  and  5,  adequately  represent  the  uncertain  situations 
that  exist  in  the  defense  acquisition  environment.  Two  of  the 
reasons  for  this  follow.  First,  the  normal  curve  assigns  some 
(very  small)  probability  to  the  occurence  of  an  event,  no  matter 


2 

In  this  section,  for  purposes  of  clarity,  the  uncertain 
variable  will  always  be  cost.  Other  variables  of  interest  to 
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how  far  to  the  right  or  left  it  might  lie.  Since  no  program  can 
have  a  negative  cost  outcome  and  because  there  generally  is  some 
dollar  value  (greater  than  zero)  that  represents  the  lowest 
possible  program  cost,  the  PDF's  normally  used  in  this  environ¬ 
ment  touch  the  horizontal  axis  somewhere  to  the  right  of  zero. 
An  example  of  this  is  shown  in  Figure  6. 


Probability 

Density 
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Figure  6 

SKEWED  PDF  FOR  ELEMENT  COST 


A  second  difference  results  from  the  fact  that,  just  by  the 
nature  of  the  defense  acquisition  business,  there  is  a  higher 
probability  of  the  ultimate  cost  outcome's  exceeding  the  most 
likely  value  than  there  is  of  the  outcome's  being  less  than  the 
most  likely  value.  If  this  is  the  case,  then  the  PDF  cannot  be 
symmetric  and  must  take  on  a  shape  like  that  shown  in  Figure  6. 
It  is  said  to  be  skewed  right.  Comparing  Figure  5  to  Figure  6, 
there  is  now  a  greater  probability  (larger  shaded  area)  that  the 
program  cost  will  be  between  $7  and  8M  than  between  $4  and  5M. 

The  amount  that  a  PDF  is  skewed,  like  the  amount  of  uncer¬ 
tainty  it  represents,  can  be  expressed  numerically,  by  a  quantity 


the  program  manager  (e.g.,  schedule  and  technical  perfor¬ 
mance)  can  be,  and  often  are,  assessed  in  the  same  way. 
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called  "skewness”.  Absolute  skewness  is  measured  in  various  ways 
by  different  researchers,  but  one  of  the  better  known  ways 
mesures  it  as  the  difference  between  the  mean  (average)  and  the 
mode  (most  likely)  values.  Dividing  this  difference  by  the  stan¬ 
dard  deviation  gives  a  relative  measure  of  skewness,  which  can  be 
positive  or  negative.  A  relative  skewness  measure  of  more  than  1 
(in  absolute  value)  is  considered  to  represent  a  highly  skewed 
PDF.  This  would  indicate  that  there  is  substantial  uncertainty 
regarding  the  original  cost  estimates,  and  that  there  is  a  higher 
probabiltiy  of  costs  exceeding  the  original  estimate  than  of 
their  falling  short. 

PDF's  have  certain  other  characteristics,  some  of  them  fami¬ 
liar,  that  a  program  manager  should  understand  to  appreciate  what 
a  risk  assessment  tells  him,  especially  when  he  wishes  to  use  the 
skewed  types  of  PDF.  The  first  of  these  is  the  mean  (or 
"average"  or  "expected")  value.  This  is  the  average  value  that 
will  occur  given  a  very  large  number  of  experiments.  Another 
important  characteristic  is  the  mode,  which  is  defined  as  the 
value  where  the  probability  curve  is  at  a  maximum.  A  symmetric 
PDF  will  have  the  mean  and  mode  occurring  at  the  same  value, 
while  a  skewed  PDF  will  have  the  mean  removed  from  the  mode 
toward  the  skewed  side  of  the  PDF,  i.e.,  the  side  with  the  long 
"tail".  Figure  6  locates  the  mean  (the  average  cost — shown 
approximately — $8.5M)  and  the  mode  (the  cost  having  the  highest 
probability  of  occurrence  at  approximately  $6M)  for  a  skewed  PDF. 

Another  characteristic  of  PDF's  is  the  amount  of  uncertainty 
they  express.  This  is  shown  graphically  by  the  amount  of  spread 
in  the  curve.  If  the  curve  is  low  at  its  peak  and  broadly 
spread,  a  great  deal  of  uncertainty  is  expressed;  if  it  is  high 
at  its  peak  and  narrow,  little  uncertainty  is  expressed.  These 
qualities  are  measured  by  expressions  called  "standard 
deviation",  (or  standard  error)  and  "coefficient  of  variation". 

The  CDF  provides  a  different  kind  of  information,  a  kind  that 
is  not  as  easy  to  visualize  by  using  a  PDF.  As  has  been  said,  in 
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a  PDF  the  probability  of  a  range  of  occurences  is  represented  as 
the  area  under  the  curve  between  two  values  bounding  that  range. 
Since  it  is  hard  to  interpret  such  a  representation,  the  CDF  is 
commonly  used.  The  CDF  shows  on  its  vertical  axis  the  area  under 
the  PDF  to  the  left  of  some  particular  value  on  the  horizontal 
axis.  Thus,  as  shown  on  the  bottom  of  page  11-11,  the  CDF  allows 
such  statements  to  be  made  as  "there  is  a  40%  chance  that  the 
cost  of  the  program  will  be  less  than  $5M."  Of  course,  one  can 
also  state  that  "there  is  a  60%  chance  that  the  cost  of  the 
program  will  exceed  $5M." 

"Confidence"  is  another  term  often  used  in  a  statistical  con¬ 
text.  For  example,  a  value  (like  cost)  may  be  said  to  lie  in  a 
given  95%  confidence  interval,  or  there  may  be  a  90%  confidence 
that  cost  will  be  at  or  below  some  value.  These  kinds  of  state¬ 
ments  can  be  made  by  using  CDF's,  as  also  shown  on  page  11-11. 
The  program  manager  can  substitute  "probability"  for  "confidence" 
in  the  latter  statement,  and  he  can  translate  the  former  to  read 
"There  is  95%  probability  that  the  cost  of  interest  is  contained 
within  the  stated  interval." 

To  summarize  the  information  contained  in  PDF's  and  CDF's,  one 
can  say  that  PDF's  provide  intuitive  information  about  the  uncer¬ 
tainty  expressed,  while  CDF's  provide  more  specific,  but  limited, 
quantification  of  some  information  not  readily  retrievable  from 
PDF's.  A  PDF  gives  the  limits  expected  to  contain  the  uncertain 
value  (the  left  and  right  end-points  of  the  PDF).  It  shows  the 
most  likely  value  (the  mode),  and  it  may  show  the  average  value. 
It  shows  whether  the  value  is  considered  more  likely  to  exceed  or 
fall  short  of  the  most  likely  value  by  whether  it  has  positive 
(right-hand)  skewness  or  negative  (left-hand)  skewness.  Lastly, 
it  provides  a  feel  for  the  uncertainty  expressed,  by  whether  it 
is  narrow  and  peaked  (low  uncertainty)  or  flat  and  wide  (high 
uncertainty) .  Numerical  measures  of  these  visual  signs  are  often 
provided  for  comparative  purposes.  The  CDF  allows  statements  to 
be  made  regarding  the  probability  of  the  outcome  being  less  than 
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(or  exceeding)  some  particular  value.  CDF's  also  allow  confidence 
statements  to  be  made. 

D.  WHAT  IS  A  (FORMAL,  QUANTITATIVE)  RISK  ASSESSMENT? 


A  formal  quantitative  risk  assessment  is  a  mathematical  means 
of  integrating  the  detailed  information  possessed  by  numerous 
experts  while  preserving  the  experts'  uncertainty  and  the  complex 
relationships  between  the  elements  of  information. 3 

The  essence  of  the  result  is  a  statement  of  the  probability 
of  program  outcomes  (cost,  schedule,  technical  performance,  or 
combinations  of  them).4 

Although  these  statements  hc.ve  always  been  theoretically 
possible,  practical  difficulties  are  such  that  some  simplifying 
assumptions  are  always  required,  and  it  is  the  type  and  amount  of 
simplification  that  constitute  the  reasons  for  there  being  many 
risk  assessment  techniques.  In  most  comprehensive  types  of  risk 


3The  reader  who  is  looking  for  an  analysis  that  solves  his 
problem  of,  say,  cost  risk,  in  the  same  way  that  an  engi¬ 
neering  method  solves  a  design  problem,  should  recall  that  no 
probability  statement  (read  "risk  assessment")  can  say  what  a 
program  will  cost;  it  only  represents  the  aggregated  opinions 
of  the  experts  as  to  what  it  will  probably  cost.  Also,  there 
is  no  proof  of  correctness  in  the  final  outcome's  lying 
closer  to  the  value  given  by  the  risk  analysis  at  some  per¬ 
centage  level  (typically  a  high  one)  than  to  the  original 
point  estimate.  All  that  can  be  said  about  the  point  esti¬ 
mate  is  not  that  it  was  wrong,  but  that  as  an  opinion  of  risk 
it  failed  to  represent  the  thinking  of  the  experts  very  well. 
As  far  as  the  outcome  goes,  it  only  represents  one  "throw  of 
the  dice"  in  a  ( never-to-be-undertaken)  infinite  series  of 
throws,  the  record  of  which  will,  if  the  experts  are  right, 
be  closer  and  closer  to  the  PDF  resulting  from  the  risk 
assessment.  It  is  no  more  a  proof  of  correctness  than  is  any 
other  value.  Seven  may  be  the  most  likely  outcome  of  a  throw 
of  a  fair  pair  of  dice,  but  the  actual  occurrence  of  a  seven 
does  not  justify  the  prediction  of  a  seven  on  any  single 
throw,  although  it  may  justify  a  wager  on  the  occurrence  of 
the  seven. 

4A  PDF  or  CDF  relating  to  two  or  more  outcomes  and  their 
occurrence  together  is  called  a  "joint"  PDF  or  CDF.  Since 
three  dimensions  are  involved,  it  is  usually  not  graphed. 
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assessment,  the  statement  of  program  outcome  probability  will  be 
made  in  the  form  of  a  PDF,  a  CDF,  and  the  numerical  measures 
described  above.  In  other  types  of  less  depth,  the  results  are 
restricted  amounts  of  the  same  kinds  of  information  or  are  simply 
determinations  of  budget  levels  needed  to  accommodate  risk  to 
some  degree. 
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CHAPTER  III 


WHY  CONSIDER  RISK  ASSESSMENT? 


A.  GENERAL 


The  problem  of  acquisition  program  cost  growth  seems  ever¬ 
present.  As  a  consequence,  OSD  and  military  services  have 
initiated  a  number  of  management  programs  over  the  past  three 
decades  in  efforts  to  control  cost  growth  and  to  reduce  costs  to 
more  acceptable  levels.  The  most  significant  of  the  recent 
programs  is  the  Acquisition  Improvement  Program  which  started 
with  the  Carlucci  memorandum  referenced  in  Chapter  I.  Initiative 
Number  11,  which  provides  the  stimulus  for  this  handbook,  may 
seem  to  some  program  managers  to  be  telling  them  to  do  what  they 
consider  to  be  the  primary  function  they  currently  perform,  the 
management  of  risk  reduction.  "Besides,"  they  might  say,  "my 
cost  estimates  are  based  on  the  latest  cost  estimating  techniques 
using  data  from  the  most  similar  known  programs,  and  contain  a 
reserve  for  unknowns  wherever  that  seems  reasonable."  They  feel 
that  there  can  be  no  substitute  for  intimate  program  knowledge  at 
the  top  management  level,  and  that  their  understanding  of  program 
risk  can  only  derive  from  immersion  in  the  program's  day-to-day 
financial,  engineering,  and  scheduling  problems. 

The  business  of  program  office  personnel  is  certainly  the 
continuing  analysis  and  modification  of  plans,  designs,  and  cost 
estimates,  and  the  intensity  of  this  effort  is  not  lessened  by 
employing  more  formalized  methods  of  risk  assessment  than  the 
somewhat  intuitive  approaches  in  customary  use.  Indeed,  this 
intense  study  of  the  program  is  a  prerequisite  for  the  use  of 
more  formalized  methods  and  in  fact  is  the  first  step  in  a  quan¬ 
titative  risk  assessment. 

What,  then,  does  formal  quantitative  risk  assessment  provide 
that  the  customary  "problem  immersion"  approach  does  not?  The 
immediately  discernable  benefit  is  the  one  common  to  all  for- 
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malized  analysis  techniques;  i.e.,  provision  of  an  accepted 
structure  of  logic,  permitting  improved  communication  and  eva¬ 
luation.  A  formalized  logical  structure  enhances  identification 
of  points  of  agreement  and  thus  promotes  objectivity. 

The  next  most  obvious  benefit,  derived  from  quantification, 
is  also  again  an  improved  communication,  this  time  in  identifying 
the  relative  importance  of  assumptions  and  program  elements  and 
the  degree  of  any  existing  disagreements.  Quantification  futher- 
more  permits  investigation  of  the  sensitivity  of  assessment  con¬ 
clusions  to  the  identified  disagreements.  A  final  general  advan¬ 
tage  of  formalized  quantitative  risk  assessment  over  unstructured 
approaches  is  its  capability  to  utilize  expert  opinion  directly 
and  in  a  clearly  identifiable  and  measurable  way.  The  limita¬ 
tions  of  the  human  mind  are  such  that  unstructured  assessments  of 
risk  are  unlikely  to  account  for  more  than  six  or  eight  separate 
considerations,  and  the  amount  of  modification  to,  or  weighting 
of,  expert  opinion  is  not  likely  to  be  discernable,  thus,  basis 
for  objective  discussion  may  fail  to  exist. 

A  program  manager  finds  himself  drawn  in  two  directions 
regarding  budgets.  If  he  fails  to  obtain  sufficient  funds  on 
time,  he  risks  either  program  failure  or  cost  growth  due  to 
program  stretches  or  inefficiencies.  If  he  requests  excessive 
funds  he  risks  program  cancellation  for  non-affordability.  A 
formalized  quantitative  risk  assessment  permits  him  to  determine 
otherwise  unknown  implications  of  funding  levels,  to  communicate 
those  implications  credibly  to  reviewing  authorities,  and  thus  to 
enhance  his  chance  for  program  success. 

B.  COST  VS.  BENEFIT 

1.  Costs . 

If  cost  can  be  considered  as  the  expenditure  of  resources,  a 
program  manager  will  be  concerned  with  at  least  four  "costs"  of 
conducting  a  risk  assessment;  i.e.,  expenditure  of  funds,  time, 
manpower,  and  management  attention. 
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The  funds  required  to  conduct  a  risk  assessment  can  vary  from 
small  to  substantial  amounts,  although  compared  to  the  costs 
avoided  as  a  result  of  a  risk  assessment  it  is  probably  negli¬ 
gible.  Nevertheless  the  program  manager  will  be  concerned  about 
even  moderate  expenditures,  for  example,  early  in  a  program  when 
overall  cash  flow  is  low. 

Time  can  be  considered  a  "cost"  in  that  delay  pending  comple¬ 
tion  of  a  risk  assessment  can  forestall  accomplishment  of  other 
program  activities.  R.i  sk  assessment  techniques  vary  in  the  time 
required  from  a  few  hours  to  a  number  of  months,  but  the  selec¬ 
tion  of  techniques  and  their  integration  and  program  activities 
can  be  accomplished  so  as  to  minimize  schedule  impact. 

Manpower  as  a  "cost"  includes  the  man-months  required  of 
subordinate  program  office  personnel  or  of  support  office  ana¬ 
lysts.  These  "costs"  may  vary  from  a  few  man-hours  to  more  than 
six  man-months.  Again,  techniques  are  available  which  allow 
selection  of  a  method  appropriate  to  the  resources  of  the  program 
office. 

A  final  "cost"  is  that  of  management  attention.  It  has  been 
found  that  for  a  program  manager  to  derive  significant  decision¬ 
making  benefit  from  a  risk  assessment,  he  must  be  willing  to  com¬ 
mit  substantial  amounts  of  personal  attention  tc  the  assessment. 
He  must  understand  and  approve  the  concept  and  approve  the 
assumptions  of  any  model  developed  by  analysts.  He  must  review 
the  analysts'  progress  and  findings,  keeping  them  updated  on 
program  changes.  More  detailed  discussions  and  estimates  of 
these  "costs"  can  be  found  in  Chapter  V. 

2.  Benefits 


The  expenditure  of  the  resources  cited  above  yields  direct 
and  indirect  benefits,  the  direct  being  the  explicit  output  of 
the  risk  assessment,  and  the  indirect  being  by-products  of  the 
assessment  process. 
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Direct  benefits  include  credible  budgets  with  explicit 
(although  probabilistic)  consequences.  Since  in  the  case  of 
detailed  analyses,  PDF's  and  CDF's  can  be  examined,  a  program 
manager  can  see  the  amount  of  risk  he  is  facing  (by  such  measures 
as  standard  deviation  and  coefficient  of  variation)  and  whether 
his  program  is  more  likely  to  underrun  or  overrun  (by  the  sign  of 
the  skewness)  and  the  relative  magnitude  of  the  likelihood  (by 
the  magnitude  of  skewness).  He  can  also  see  the  most  likely  cost 
(by  the  magnitude  of  skewness  —  the  mode  and  the  probability 
that  any  funding  level  will  be  sufficient  (the  probability  asso¬ 
ciated  with  that  level  on  the  CDF),  besides  the  probability  that 
costs  will  be  within  any  specified  limits  (the  CDF  probability  of 
the  lower  subtracted  from  that  of  the  higher).  Needless  to  say, 
material  in  support  of  credible  estimates  will  aid  in  this  justi¬ 
fication  of  costs  used  in  service  POMs,  DOD  budget  hearings,  and 
Congressional  reviews. 

An  assessment  of  schedule  risk  can  provide  direct  evaluation 
of  alternative  schedules,  including  not  only  their  probabilities 
of  success,  but  also,  their  probable  funding  requirements.  | 

Examination  of  the  inputs  to  risk  assessments  identifies  the  j 
high  risk  program  elements  and  thus  points  out  the  areas  where 
risk  management  should  be  focused.  4 


One  indirect  benefit  which  is  a  significant  by-product  of  a 
detailed  risk  assessment  is  the  improvement  of  program  definition 
resulting  from  the  methodological  requirement  to  specify  all 
program  activities  and  their  dependency  on  other  elements.  Not 
only  is  program  definition  improved  but  all  members  of  the 
program  team  develop  improved  understandings  of  the  total  program 
and  their  places  within  it. 

A  second  indirect  benefit  is  the  enhanced  problem  solving 
insight  provided  by  knowledge  of  how  risks  and  performances  can 
vary  when  conditions  and  assumptions  change. 
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The  list  of  specific  decision  categories  supportable  by 
application  of  quantitative  risk  assessment  techniques  (some  of 
which  have  not  been  mentioned  previously)  include: 

o  Program  control 

o  Major  planning  decisions 

o  Acquisition  strategy  selection 

o  Technical  alternative  selection 

o  Contract  structuring 

o  Source  selection 


Discussion  of  how  quantitative  risk  assessment  supports  de¬ 
cisions  of  these  types  can  be  found  in  Chapter  V  and  Appendix  D. 
Appendix  L  summarizes  in  chart  form  the  products  of  various 
assessment  methods  and  matches  them  against  decision  categories 
to  which  they  can  be  applied. 


CHAPTER  IV 


METHODS  OVERVIEW 


A.  GENERAL. 

Although  researchers  continue  to  develop  new  risk  assessment 
techniques  and  applications,  this  chapter  will  summarily  describe 
only  the  seven  quantitative  techniques  in  predominant  current 
use.  The  quantitative  techniques  to  be  described  are: 

o  Network  analysis 

o  The  method  of  moments 

o  Decision  analysis 

o  WBS  simulation 

o  Graphics 

o  Estimating  relationships 

o  Risk  factors 

Although  there  are  applications  mixing  the  techniques,  these 
seven  constitute  the  majority  of  those  potentially  applicable  to 
ongoing  (rather  than  conceptual)  programs. 

In  addition  to  the  quantitative  techniques,  there  are  two 
non-quantitative  methods  which  in  this  handbook  will  be  called 
"structured  qualitative"  and  "engineering  analysis".  The  latter 
is  this  handbook's  term  for  the  unstructured  "problem-immersion" 
approach  discussed  in  Chapter  III;  the  former  is  a  more  struc¬ 
tured  derivative.  These  two  are  mentioned  because  "engineering 
analysis"  is  the  approach  in  predominant  military  acquisition 
program  use  and  the  "structured  qualitative"  method  (exemplified 
by  [53)  represents  an  upgrade  of  the  "engineering  analysis" 
method.  The  "structured  qualitative"  approach  improves  on 
"engineering  analysis"  by  providing  explicit  criteria  for  judg¬ 
ment  of  risk,  by  adopting  a  logical  structure,  and  by  docu¬ 
menting  its  rationale. 
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The  seven  quantitative  risk  assessment  methods  which  follow 
are  described  briefly  in  the  following  sections,  and  discussed  in 
greater  detail  in  Appendix  F. 

B.  NETWORK  METHODS 

1.  General 

Most  managers  today  are  familiar  with  the  concept  of  modeling 
an  acquisition  program  as  a  network,  such  as  that  in  Figure  7, 
made  popular  by  the  Polaris  submarine  project  and  termed  the 
Program  Evaluation  and  Review  Technique  (PERT). 


Figure  7 

PROGRAM  REPRESENTED  AS  A  NETWORK 

In  such  a  network,  each  circle  represents  a  decision  point  or 
milestone,  and  each  line  represents  an  activity  that  must  be 
finished  to  advance  the  programf  that  consumes  resources,  or  that 
takes  time.  In  the  PERT  approach  the  objectives  were  to  manage 
schedule  risk  by  establishing  the  shortest  development  schedule?,, 
to  monitor  and  project  progress,  and  to  fund  or  apply  necessary 
resources  for  maintaining  the  schedule.  Successors  to  PERT 
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included  PFRT/COST  in  which  the  minimum  cost  path  through  the 
network  was  estimated,  and  the  project  was  managed  to  minimum 
cost.  PERT  models  lacked  the  capability  to  include  probability 
information  on  the  cost  and/or  schedule,  however  current  network 
models  can  use  computers  to  arrive  at  probability  statements 
incorporating  evaluations  of  cost,  time,  or  performance  level 
values  by  means  of  a  technique  called  simulation. 

Typically,  the  analysis  in  PERT  models  was  based  on  average, 
and  sometimes  extreme,  values  of  activity  duration.  In  current 
techniques,  1-he  network  is  defined,  and  the  activity  probabili¬ 
ties  of  cost  or  schedule  are  described.  Then,  by  using  computers 
to  simulate  a  large  number  of  program  completions  in  which  events 
occur  according  to  their  probabilities,  the  probabilities  are 
evaluated  for  the  network  as  a  whole.  Subsequently,  probability 
expressions  such  as  confidence  levels,  etc.  (as  described  in 
Chapter  II)  can  be  made.  Thus  the  subtle  and  complex  interrela¬ 
tionships  of  the  hundreds  of  activities  required  to  complete  a 
program  can  contribute  to  the  same  degree  they  would  in  the 
actual  program,  assuming  all  activities  truly  behave  as  expected. 
The  fact  that  the  activities  may  not  behave  exactly  as  expected, 
that  is,  that  no  model  is  perfect,  is  irrelevant.  The  decisions 
must  be  made  and  must  be  based  on  the  best  information  available 
at  the  current  time.  The  model  represents  that  best  available 
information,  or  the  analyst  must  change  the  model. 

2 .  Inputs/Outputs 
a.  Inputs 


Establishment  of  a  network  model  requires  definition  of 
each  program  activity,  its  beginning,  its  end,  its  possible  pre¬ 
decessors,  and  its  possible  successors,  at  some  specified  level 
of  detail.  The  last  phrase  signifies  that  activities  themselves 
can  almost  always  be  restructured  into  more  detailed  activity 
networks.  Part  of  the  art  of  network  modeling,  a  matter  the 
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program  manager  may  wish  to  review,  is  the  level  of  detail  to 
use,  since  increased  detail  increases  complexity,  cost,  time,  and 
manpower  requirements.  The  inputs  described  so  fai  say  nothing 
about  probability.  Each  activity  must  also  be  described  in  terms 
of  probability,  with  the  probabilities  relating  to  cost,  sche¬ 
dule,  manpower,  technical  level  of  achievement,  or  some  com¬ 
bination.  Also,  when  it  is  not  certain  which  activities  can 
occur  at  a  decision  point,  it  is  necessary  to  input  probabilities 
of  those  activities.  Some  computer  programs  limit  the  charac¬ 
teristics  that  can  be  described  by  probabilities,  while  some  are 
virtually  unlimited  but  require  more  computer  resources  (see 
Chapter  V) . 


b.  Outputs 


Network  model  outputs  typically  include  PDF's,  CDF's, 
and  the  statistical  measures  described  in  Chapter  II  for  the  cost 
or  schedule  of  the  entire  program  or  specified  parts  of  it. 
Other  outputs  include  lists  of  the  activities  or  groups  of  acti¬ 
vities  that  have  caused  program  difficulty,  in  order  of  their 
frequency  of  causing  it.  This  is  called  "criticality,"  a  term 
derived  from  the  PERT  term  "critical  path".  A  PERT  critical  path 
is  the  sequence  of  activities  in  a  network  having  the  longest 
duration  or  highest  cost. 


3.  Resource  Requirements 


Since  most  network  risk  assessments  accomplished  in  the  DOD 
are  carried  out  by  functional  support  offices,  risk  assessment 
dollar  costs  should  be  estimated  from  manpower  requirements.  A 
comprehensive  network  analysis  for  a  major  program  may  require 
definition  of  between  200  and  1000  activities  and  require  2  to  6 
man-months  of  GS-12  to  GS-14  analyst  effort  for  gathering 
experts’  PDF's  and  for  building  the  network.  The  analysts  must 
obtain  lists  of  program  activities  and  information  on  the 
sequence  of  activities  from  Program  Management  Office  (PMO)  per- 
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sonnel  and  from  the  members  of  the  organizations  supporting  the 
PMO.  Obtaining  this  information  consumes  more  time  and  requires 
more  re-chec.king  than  might  seem  necessary.  This  is  because  the 
program  plan  is  usually  under  continual  revision  and  definition, 
and  the  support  personnel  themselves  do  not  fully  understand  the 
program  activity  interdependencies. 

Although  the  difficulty  and  time  required  for  network  defini¬ 
tion  is  a  problem,  the  effort  of  constructing  a  consistent  and 
acceptable  network  model  forces  the  responsible  participants  to 
plan  more  effectively  and  to  understand  how  their  own  segments  of 
the  program  fit  into  the  whole.  Program  managers  have  indicated 
that  this  benefit  can  justify  all  the  effort  for  accomplishment 
of  a  formal  network  risk  assessment. 

Having  recognized  the  difficulties  associated  with  developing 
networks  for  particular  acquisition  programs,  some  Army  commands 
(at  least  MERADCOM  and  CECOM)  have  generated  "dictionaries"  of 
the  activities  typically  required  during  the  life  cycle  of  a 
program.  These  "dictionaries"  provide  descriptions  of  the  acti¬ 
vities,  typical  durations,  and  interrelationships  with  other 
activities.  The  activity  descriptions,  along  with  a  sample  net¬ 
work,  provide  an  excellent  starting  point  for  PMO  personnel  and 
analysts  attempting  to  develop  a  program  network.  The  existing 
"dictionaries"  are  somewhat  tailored  as  they,  of  course,  reflect 
service  and  command-peculiar  requirements  and  apply  to  the  types 
of  systems  procured  by  the  author  commands.  They  are  mentioned 
here  as  suggestions  of  what  would  be  possible  in  other  commands. 

4 .  Applications 

Network  risk  assessment  models  have  been  used  throughout  the 
Army  in  support  of  In  Process  Review  (IPR),  Army  Systems 
Acquisition  Review  Council  (ASARC),  and  Defense  Systems 
Acquisition  Review  Council  (DSARC)  program  reviews.  Navy  program 
offices  of  NAVSEA  have  obtained  contractor  support  for  network 


based  risk  assessments  for  appraisal  and  control  of  both  major 
acquisition  programs  and  overhaul  programs.  No  examples  of  USAF 
use  of  network  risk  assessment  techniques  were  found  during  pre¬ 
paration  of  this  handbook. 

C.  Decision  Analysis  Methods 


1.  General 


Decision  analysis  is  the  examination  of  decisions  by 
brea*.  ng  them  into  the  sequences  of  supporting  decisions  and  the 
resulting  uncertain  occurrences.  Usually,  these  sequences  are 
represented  as  decision  "trees"  such  as  that  in  Figure  8. 
Interesting  and  extensive  treatment  of  the  subject  may  be  found 
in  [16]. 
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DECISION  TREE 

The  first,  left-hand  square  in  Figure  8  represents  an 
aircraft  conceptual  decision  that  can  be  made  in  either  of  two 


ways;  i.e.,  select  a  tilt-wing  design  or  a  helicopter.  The 
circles  that  follow  represent  test  programs  that  have,  on  the  top 
branch,  three  possible  outcomes  or,  on  the  bottom  branch,  two 
possible  outcomes.  The  likelihood  of  each  of  these  outcomes  is 
shown  on  the  appropriate  branch  (e.g.,  P=.3  indicates  a  probabi¬ 
lity  of  3  in  10  that  the  tilt  wing  design  will  fail  to  meet  test 
objectives).  The  next  set  of  squares  represent  decisions  that 
might  have  to  be  made  after  each  outcome.  Some  of  these  deci¬ 
sions  are  followed  by  new  tests.  In  the  boxes  on  the  right-hand 
side  of  the  tree,  the  funding  consequences  of  each  of  the 
branches  are  shown. 

The  complication  of  managing  a  large  and  unwieldly  decision 
tree  can  be  avoided  by  using  a  simplified  approach,  called 
Probabilistic  Event  Analysis  (PEA)  and  described  in  [20].  While 
the  simplified  method  provides  for  determining  which  activities 
and  milestones  for  major  WBS  elements  contribute  to  cost,  as 
though  milestones  were  decisions  on  a  decision  tree,  the  tree 
does  not  have  to  be  drawn,  but  the  logic  is  the  same.  The  ana¬ 
lyst  defines  the  problems  which  he  feels  may  effect  the  program 
cost  or  schedule  and  assesses  the  likelihoods  of  their 
occurrence.  The  analyst  continues  by  obtaining  assessments  of 
the  effect  of  each  type  of  problem  on  the  cost  or  schedule  of  the 
relevant  function  or  milestone.  From  the  probabilities  and  the 
magnitude  of  effects,  an  "expected  value"  of  cost  increment  for 
each  effort  can  be  calculated. 

A  more  elaborate  method  is  described  in  [8],  which  first 
orients  the  assessment  around  a  specific  program  sequence  that 
allows  the  analyst  to  define  the  program  as  in  a  decision  flow 
diagram  (decision  tree).  In  this  method,  time  and  cost  effects 
of  identified  problems  are  estimated  as  in  the  former  method.  A 
series  of  simple,  but  tedious  and  voluminous,  probability  calcu¬ 
lations  allow  the  analyst  to  establish  cost  and  schedule  probabi¬ 
lity  distributions  for  each  phase.  These  distributions  are  then 
combined  by  using  them  as  inputs,  not  to  a  pencil  and  paper 
calculation,  but  to  a  simulation  on  a  computer. 
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Although  the  mathemat:  js  underlying  these  decision  tree 
approaches  are  undemanding  and  the  network  simulation  can  be  much 
simple^  than  those  discussed  under  network  methods,  analyzing  the 
relationships  is  not  simple,  and  any  computer  use  requires  a 
degree  of  sophistication  to  carry  out  the  programming.  It  is 
particularly  difficult  to  ensure  comprehensive  identification  of 
problems  and  time/cost  effects. 

2 .  Input/Output 

a.  Inputs 

Decision  analysis  methods  require  definition  of  the 
program  decisions  to  be  made,  again,  as  in  any  model,  at  some 
convenient  level  of  detail.  These  may  include  many  of  the 
milestones  that  are  inputs  to  a  network  analysis,  but  in  the 
decision  analysis  model,  the  emphasis  is  on  the  points  where 
decisions  are  made,  not  on  the  sequencing  of  activities.  Rather 
than  defining  PDF's  for  activities,  usually  a  specific  small 
number  of  problem  occurrences  is  defined  for  any  point  in  the 
decision  logic  where  chance  enters  in,  and  the  subject  matter 
experts  are  polled  to  obtain  probability  estimates  for  all 
possible  occurrences,  the  problems  that  could  ensue,  their  cost 
impacts,  and  their  probabilities.  Thus,  the  inputs  for  this 
method  are: 

DECISIONS 

Potential  Probability  Possible  Probability  Cost 

Problems  of  Occurrence  Resolutions  of  Success  Impact 

In  the  method  of  18],  not  only  are  the  problems,  their  probabi¬ 
lities  and  their  cost  (or  schedule)  impacts  obtained,  but  each 
problem's  resolutions,  resolution( s )  probabilities  of  success, 
and  resolution  cost  (or  schedule)  impacts  are  also  used. 

b.  Outputs 

The  cost  oriented  Probabilistic  Event  Analysis  output  is 
a  reserve  to  be  added  to  the  baseline  budget  established  without 
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consideration  of  the  problems  subsequently  addressed  in  the  PEA. 
Nothing  similar  to  a  PDF  is  generated  and  no  statement  can  be 
made  relative  to  the  probability  of  exceeding  the  new  budget. 


The  method  described  in  [8]  provides  "joint"  probability 
graphs  showing  program  probability  of  success  at  given 
probabilities  of  needing  stated  levels  of  funding  (or  time). 

3 .  Resource  Requirements 

Probabilistic  Event  Analysis  was  developed  to  demand  less 
time  and  analytical  skill  than  network  methods.  As  with  other 
methods,  the  most  time-consuming  part  of  a  PEA  is  probably 
obtaining  comprehensive  determination  of  critical  occurrences, 
their  probabilities  and  their  impacts,  in  other  words,  data 
collection.  For  a  program  for  which  the  plans  have  been  stabi¬ 
lized,  a  GS-12  or  -13  might  require  no  more  than  a  few  days  to 
complete  a  PEA. 

The  type  of  decision  analysis  described  in  [ 8 j  will  probably 
require  an  analyst  of  greater  knowledge  and  skill  than  a  PEA  and 
perhaps  than  even  a  network  assessment.  Some  computer 
programming  will  be  required,  data  collection  is  likely  to  be 
more  comprehensive  and  demanding,  and  interpretation  of  results 
may  require  more  sophistication.  A  typical  team  (that  for  [15]) 
consisted  of  a  GS-15  analyst  with  two  lower  grade  (GS-13/14)  team 
members,  and  the  assessment  consumed  about  4  months. 

4.  Applications 

No  examples  of  application  of  the  PEA  method  were  found  in 
preparation  of  this  handbook.  The  method  described  in  [8]  has 
been  used  at  the  U.S.  Army  Air  Mobility  R&D  Laboratory,  Ames 
Research  Center,  Moffet  Field,  CA.,  for  a  Cobra  Armament  Program, 
and  for  the  XV-15  program.  No  USN  or  USAF  applications  of  deci¬ 
sion  analysis  applications  to  risk  assessment  were  found  during 
handbook  preparation. 


V 


D,  THE  METHOD  OF  MOMENTS 


1.  General 


In  network  risk  assessment  methods,  the  program  activities 
are  related  according  to  their  sequencing,  and  the  combined 
effects  of  their  probabilities  are  determined  by  simulation.  In 
the  method-of -moments  (developed  in  [13])  no  network  sequence 
relations  are  defined,  and  the  combined  effects  of  program  ele¬ 
ment  probabilities  are  determined  by  mathematics  instead  of  simu¬ 
lation.  Method-of -moments  assessments  concern  themselves  with 
cost  risk  and  proceed  from  a  framework  of  costs  like  a  Work 
Breakdown  Structure  [14],  sometimes  called  a  cost  breakdown. 
Each  of  the  cost  elements  (not  activities,  as  was  the  network 
case)  has  some  probability  "statement"  (PDF)  associated  with  it. 
PDF's  may  be  humped  and  symmetric,  indicating  that  cost  is 
equally  as  likely  to  be  above  as  below  the  most  likely;  or  humped 
but  skewed,  indicating  a  greater  likelihood  that  cost  will  be  on 
one  side  of  the  most  likely  value  than  the  other.  However  they 
are  shaped,  they  combine  to  produce  some  total  program  cost  PDF. 


As  was  stated  in  paragraph  D  of  Chapter  II,  simplifying 
assumptions  are  required  in  formulation  of  any  mathematical 
description  (model).  For  example,  one  of  the  simplifications 
used  in  network  models  is  selecting  the  level  of  detail.  In  the 
method-of -moments ,  a  major  simplification  used  is  that  of 
selecting  a  total  combined  PDF,  which  is  assumed  to  be  a  type 
possible  to  calculate  by  this  technique.  While  this  is  a  major 
assumption,  it  does  not  render  the  technique  worthless. 
Typically  the  assumed  PDF  exhibits  right-hand  (positive) 
skewness,  is  assumed  to  have  some  unspecified  lowest  possible 
value  (hit  the  axis  on  the  left),  and  may  or  may  not  have  a 
highest  possible  unspecified  value  (hit  the  axis  on  the  right). 
Typically,  a  network  simulation  cost  risk  analysis  does  not  pro¬ 
duce  one  of  these  "well-behaved"  PDF's,  and  some  analysts  would 
say  that  shows  that  assuming  a  well-behaved  PDF  type  makes  the 
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method-of -moments  valueless.  It  must  be  remembered,  however, 
that  many  less-than-ma jor  programs  are  unable  to  command  the 
resources  for  a  network  risk  assessment,  so  a  method-of -moments 
assessment,  while  less  than  perfect,  can  provide  otherwise  un¬ 
available  insights. 

The  method's  name  comes  from  the  fact  that  the  "moments"  of 
the  cost  element  PDF's  are  used  to  form  the  ’moments"  of  the  com¬ 
bined  PDF  of  total  cost.  Some  of  the  better  known  moments  are 
the  mean  and  the  variance  (the  standard  deviation  squared),  which 
describe  respectively  the  "center  of  gravity"  of  the  distribution 
and  the  spread  of  costs  around  the  mean.  There  are  other 
moments,  such  as  "skewness",  which  describes  the  distribution's 
assymetry,  and  others  more  difficult  to  visualize.  For  more  on 
moments  themselves,  reference  should  be  made  to  a  probability 
theory  or  statistical  theory  textbook. 

s 

A  PDF  is  mathematically  described  by  a  formula  having  certain 
constants  (parameters),  the  values  of  which  establish  the  PDF's 
shape.  The  method-of -moments  correctly  gives  the  parameters  that 
determine  where  the  PDF  starts,  where  it  peaks,  how  much  it 
spreads,  how  quickly  it  comes  down  off  the  peak,  and  where  it 
ends.  The  procedures  are  discussed  in  more  detail  in  Chapter  V. 

In  theory,  even  without  the  method-of -moments,  one  could  com¬ 
pute  a  resultant  PDF  describing  the  multiplication  and  addition 
of  a  number  of  uncertain  element  costs.  As  a  practical  matter, 
though,  it  has  been  considered  heretofore  impossible,  even  when 
using  large  computers,  which  is  why  analysts  have  usually  had  to 
resort  to  simulation.  Given  the  met hod-of -moments ,  analysts  and 
managers  can  now  make  inferences  about  program  cost  probabilities 
with  substantially  less  effort  than  a  network  analysis  might 
require.  For  example,  it.  is  possible  to  carry  out  the  method-of - 
moments  calculations  on  a  hand  calculator,  although  appropriate 
programs  would  have  to  be  written  to  make  it  convenient. 
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2. 


Input/Output 

a.  Input 

As  stated  above ,  the  framework  for  a  method-of -moments 
analysis  is  the  program  cost  breakdown  rather  than  a  description 
of  program  activities.  The  cost  element  probabilities  are 
described  using  PDF's  just  as  the  activities  in  the  network 
method  are.  Because  some  of  the  cost  elements  may  be  costs  for 
exactly  the  same  activities  as  are  described  in  network 
assessments,  some  of  the  same  PDF's  may  appear,  but  the  overall 
set  of  inputs  will  probably  be  quite  different  since  in  the 
method-of -moments  the  effect  of  schedule  cannot  be  treated  inde¬ 
pendently. 

The  analyst  will  perform  calculations  using  information  taken 
from  the  input  PDF's  supplied  by  subject  matter  experts.  This 
allows  him  to  calculate  the  specific  shape  of  the  assumed  PDF  type 
representing  the  combination  of  cost  elements  into  a  total  cost. 

b.  Outputs 

Method-of -moments  assessments  directly  produce  the 
numerical  measures  associated  with  PDF's  that  were  described  in 
Chapter  II;  i.e.,  mean,  standard  deviation,  and  skewness.  From 
these,  and  using  the  analyst-assumed  PDF  type,  the  specific  PDF 
and  its  CDF  can  be  drawn.  This  permits  the  making  of  all  the 
same  kinds  of  statements  (except  those  concerning  criticality)  as 
can  be  made  following  a  network  risk  assessment  that  addresses 
cost  only. 

3.  Resource  Requirements 

The  avoidance  of  the  need  for  computers  means  that  the 
method-of-moments  could  be  carried  out  by  a  single  person  (a 
GS-13  analyst,  with  some  reading  and  study).  As  with  the  network 
method,  a  large  part  of  his  time  would  be  consumed  with  obtaining 
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experts'  PDF's,  but  definition  of  a  cost  breakdown  may  take  much 
less  time  than  definition  of  a  detailed  program  network.  For  the 
program  of  less-than-ma jor  size,  for  which  this  method  seems  best 
suited,  assuming  a  cost  breakdown  has  already  been  constructed, 
the  computations  might  require  less  than  a  week  of  effort.  If 
pre-written  programmable  hand  calculator  programs  can  be 
obtained,  only  a  few  hours  will  be  needed  to  organize  data,  per¬ 
form  calculations  and  document  numerical  output.  The  reader 
should  note  that  obtaining  experts'  PDF's  can  require  an  hour  or 
more  per  expert,  thus,  the  total  time  saved  is  not  in  the  same 
proportion  as  that  saved  in  computation.  In  other  words,  cutting 
computation  time  in  half  does  not  cut  assessment  time  in  half  if 
data  collection  time  is  the  same. 


4.  Applications 


No  examples  of  DOD  method-of -moments  application  were  found 
during  preparation  of  this  handbook. 

E.  WBS  Simulation  Methods 

1.  General 

WBS  (or  cost  breakdown)  simulation  methods  perform  the  same 
function  and  use  the  same  concepts  as  the  method-of -moments .  As 
with  the  method-of -moments ,  more  than  one  variant  exists  and 
three  are  described  in  [1],  [9]  amd  [17].  The  difference  between 
the  method-of -moments  and  WBS  simulation  lies  in  the  fact  that 
the  WBS  simulation  requires  a  computer  and  the  method-of -moments 
does  not.  The  computer  is  used  to  generate  the  PDF  for  system 
total  cost  by  performing  large  number  of  simulation  runs.  These 
runs  provide  sums  of  elements  costs  that  are  generated  from  the 
input  cost  element  PDF's.  Specific  computer  programs  that  have 
been  used  with  this  method  are  described  in  [1],  [9]  and  [16]. 
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Inputs  are  the  same  as  for  the  method-of -moments,  i.e.. 


PDFs  are  obtained  for  each  cost  element  by  consulting  experts  in 
those  element  categories. 

b.  Output 

Outputs  are  the  same  as  for  the  method-of -moments. 

3 .  Resource/Requirements 

Skill  requirements  and  costs  for  this  method  may  be  slightly 
greater  than  for  method-of -moments  assessments  since  use  of  a 
computer  is  required.  Otherwise,  resource  requirements  should  be 
nearly  equal. 

4 .  Application 

The  WBS  simulation  method  is  used  in  the  Office  of  the  Chief 
of  Naval  Operations.  A  variant  of  the  method  is  used  in  the 
Directorate  of  Cost  Analysis,  Deputy  for  Comptroller,  Armament 
Division,  Eglin  AFB ,  FL.  A  straightforward  application  is  used 
in  the  Directorate  of  Cost  and  Management  Analysis,  Comptroller, 
Headquarters  Air  Force  Systems  Command,  Andrews  AFB,  D.C. 

F.  THE  GRAPHIC  METHOD 

1 .  General 

The  graphic  method  uses  graphs  of  program  cost  element  CDF’s 
and  some  simple  algebra  to  express  program  cost  uncertainties  and 
to  combine  the  element  CDF's  to  obtain  the  resultant  overall  cost 
CDF.  That  graph  allows  the  same  statements  to  be  made  about  pro¬ 
babilities  of  cost  as  does  any  CDF.T- 


^Another  graphic  method  uses  graphs  to  determine  the  moments 
to  be  used  in  a  method-of -moments  approach.  It  is  discussed 
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This  method,  which  is  described  in  [10]  and  is  discussed  in 
more  detail  in  Appendix  F,  describes  the  uncertainty  of  Work 
Breakdown  Structure  element  costs  in  terms  of  normal 
( beli-shaped)  CDF  segments  and  from  them  finds  the  CDF  of  total 
cost  using  normal  curves.  In  other  words,  it  performs  the  same 
function  as  the  method-of -moments  and  WBS  simulation  but  makes 
use  of  different  simplifying  assumptions.  It  also  has  a  more 
restricted  result.  The  simplifying  assumption  is  that  CDF's  can 
be  approximated  by  joining  parts  of  normal  CDF's.  The  assumption 
greatly  weakens  the  validity  of  the  method  but  may  be  acceptable 
for  a  speedy  approximate  solution. 

A  special  tool  needed  for  this  method  is  "normal  probability 
paper" — a  special  graph  paper  whose  vertical  lines  are  spaced  in 
accordance  with  the  slope  of  a  CDF  for  a  normal  (bell-shaped) 
PDF.  Straight  lines  on  this  paper  have  the  values  that  a  normal 
CDF  curve  has  on  standard  praph  paper.  A  good  office  supply 
source  should  have  normal  probability  paper. 

In  this  method  the  analyst  determines  from  experts  the  cost 
estimates  for  WBS  elements  at  10  percent,  50  percent,  and  90  per¬ 
cent  confidence  levels.  These  estimates  are  the  expert's  beliefs 
that  there  are  10  percent,  50  percent,  and  90  percent  chances 
that  each  WBS  element  will  end  up  costing  less  than  or  the  same*, 
as  the  corresponding  cost. 

Having  determined  the  three  values  for  each  cost  element,  the 
analyst  next  plots  the  values  as  points  on  a  separate  piece  of 
the  normal  probability  paper  for  each  cost  element  as  in  Figure 
9. 


in  [19],  but  will  not  be  covered  further  here. 
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PROIAI I II  TT  PERCENTAGE 

Figure  9 

COST  ELEMENT  UNCERTAINTY 

For  example,  if  costs  are  in  millions  on  Figure  9,  the  plot 
shows  an  experts'  opinion  that  there  is  a  10  percent  chance  of 
costs  less  than  about  2  million  (V^g),  a  50  percent  chance  of 
costs  less  than  4  million  (V50)  and  a  90  percent  chance  of  costs 
less  than  9  million  (Vgg)  for  a  given  cost  element. 


If  V90  had  been  at  the  point  labeled  Vgg  a  straight  line 
would  have  resulted,  and  the  CDF  described  would  have  been  that 
of  a  normal  (bell-shaped)  probability.  Instead,  a  CDF  is  pro¬ 
duced  that  represents  a  PDF  like  the  darkened  lines  in  Figure  10. 


Figure  10 

EQUIVALENT  PDF  FOR  GRAPHIC  METHOD 
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Even  though  that  PDF  seems  not  to  fit  most  people's  concepts 
of  how  probability  should  behave,  it  has  two  advantages.  The 
first  is  ease  of  manipulation.  The  second  is  that  it  exhibits 
skewness,  implying  costs  are  more  likely  to  overrun  than  underrun 
which  seems  generally  more  realistic. 

With  Vio#  V50#  and  V90  from  each  cost  element  graph,  and  with 
some  factors  determined  from  graph  values  and  a  table  given  in 
Appendix  F,  the  element  cost  CDF's  can  be  combined  and  the  CDF 
graph  of  the  overall  system  cost  can  be  drawn. 

2 .  Input/Output 

a.  Input 

Input  to  this  method  is  of  the  same  type  as  that 
required  for  me thod-of -moments  and  WBS  simulation,  namely  PDF's 
representing  subject  matter  experts  estimates  of  cost  element 
probability.  These  PDF's,  however,  are  approximations  con¬ 
structed  from  the  experts'  statements  regarding  costs  at  three 
levels  of  probability,  one  of  which  must  be  50  percent,  one  less 
than  50  percent,  and  one  more  than  50  percent.  Of  course,  a  cost 
breakdown  is  also  required. 

b.  Output 

The  output  for  this  method  is  restricted  in  the  same  way 
that  decision  analysis  methods  are  limited;  the  output  is  solely 
a  CDF.  Uo  numerical  and  not  much  intuitive  information  is 
supplied  about  uncertainty  (since  there  is  no  standard  deviation 
information)  and  skewness.  What  is  provided  is  an  estimate  of 
the  most  likely  cost  (that  at  the  50  percent  level),  and  the  pro¬ 
bability  of  underrunning  (or  overrunning)  any  selected  cost. 

3.  Resource  Requirements 

Resource  requirements  for  this  method,  including  skill 
requirements,  should  be  on  the  same  order  as  those  for  an  equiva- 
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lent  niethod-of -moments  assessment.  (It  is  worth  repeating  that 
the  more  arguable  assumptions  in  this  method  suggest  that  it  pro¬ 
vides  lower  quality  results). 

4.  Applications 

No  examples  of  DOD  uses  of  this  method  were  found  during  pre¬ 
paration  of  this  handbook. 


In  this  method,  management  reserve  represents  the  amount  of 
funding,  above  that  determined  by  cost  analysis  alone,  that  it  is 
desirable  to  allocate  to  a  contract  to  provide  for  technical 
risk. 2  The  management  reserve  is  expressed  as  a  percentage  of 
the  baseline  estimate.  The  method  is  called  an  estimating  rela¬ 
tionship  method  because  it  uses  some  of  the  same  techniques  as 
the  Cost  Estimating  Relationships  (CER's)  used  in  parametric  cost 
estimating . 

Cost  Estimating  Relationships  are  the  result  of  mathematical 
methods  that  determine  a  relationship  between  cost  and  one  or 
more  measurable  system  characteristic  such  as  weight,  speed,  or 
volume.  The  method  makes  use  of  a  statistical  technique  called 
regression  analysis  to  develop  an  equation  to  fit  a  body  of  data. 
The  data  consist  of  known  costs  of  similar  systems,  along  with 
their  associated  sets  of  data  for  system  characteristics. 

The  successful  use  of  regression  analysis  for  CER's  has  led 
to  an  attempt  to  use  the  same  approach  for  estimating  budget 
requirements  to  account  for  technical  risk.  The  application  of 
this  approach  makes  use  of  categorical  definitions  for  four 
"factors"  derived  from  an  examination  of  a  number  of  contracts. 
The  "factors",  engineering  complexity,  degree  of  system  def initi- 
zation,  contractor  proficiency/experience,  and  multiple  users, 
are  qualities  having  different  levels  by  which  they  can  be 
described. 

By  using  descriptions  of  historical  contracts  and  the 
amounts  of  management  reserve  they  used,  a  formula  was  contrived 
which  is  the  basis  of  the  curve  in  Figure  11.  This  curve  should 
only  be  used  for  systems  similar  to  those  from  which  the  data 
resulted. 


2Note  that  this  method  has  been  developed  and  used  for 
contracts,  not  programs. 


To  use  the  method,  a  contract  is  described  by  program  manage¬ 
ment  personnel  and  rated  relative  to  the  factor  levels.  The  fac¬ 
tor  ratings  are  summed  and  the  result  is  entered  into  the  Figure 
11  curve  on  the  horizontal  axis  to  determine  an  appropriate  mana¬ 
gement  reserve  on  the  vertical  axis. 

2 .  Input/Output 
a.  Input 

The  inputs  to  the  model,  the  equation  of  the  curve  for 
Figure  11,  are  the  judgemental  numbers  characterizing  the 
possible  contract  factor  levels. 

b.  Output 

The  estimating  relationship  method  provides  a  percentage 
figure  to  be  applied  to  estimated  baseline  contract  cost  in 
determining  the  amount  to  supplement  the  contract  for  Management 
Reserve  and  Engineering  Change  Order  Allowance. 

3 .  Resource  Requirements 

Resource  requirements  for  this  method  match  those  for  for¬ 
mulating  the  baseline  contract  cost  estimate  plus  a  few  hours  for 
interviews  of  PMO  personnel. 

4 .  Application 

This  method  is  used  in  the  USAF  Electronics  Systems  Division. 
No  other  DOD  users  of  this  type  method  were  found  during  prepara¬ 
tion  of  this  handbook. 


H.  THE  RISK  FACTOR  METHOD 


1.  General 


The  risk  factor  method  is  a  determination  of  factors,  or 
multipliers,  by  which  to  increase  individual  program  element 


costs.  The  purpose  of  this  risk  factor  method  is  to  determine  a 
reasonable  budget,  above  that  resulting  from  a  baseline  cost 
estimate,  to  provide  for  cost  growth  anticipated  as  possible  by 
the  program  manager.  The  method  uses  a  WBS  (or  cost  breakdown) 
based  on  a  technical  breakdown  like  Figure  12  (which  is  taken 
from  [1]  but  does  not  necessarily  conform  to  [14]). 
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Figure  12 
EXAMPLE  WBS 

The  baseline  cost  estimate  is  developed  for  each  cost  ele¬ 
ment.  Applying  whatever  considerations  are  useful,  such  as  those 
discussed  in  [11],  a  risk  factor  is  established,  greater  than  1, 
for  each  cost  element.  The  estimate  element  costs  are  then 
multiplied  by  these  risk  factors  to  obtain  the  estimates  used  in 
the  budget  for  providing  a  level  of  funding  which  will  account 
for  technical  or  other  risk. 
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The  assignment  of  risk  factors  is  the  key  feature  of  this 
method  and  is  the  point  at  which  it  is  both  most  demanding  and 
has  the  least  objective  basis.  There  is  little  documented 
experience  upon  which  the  analyst  can  draw  in  order  to  substan¬ 
tiate  his  factor  assignment.  Since  the  application  is  multipli¬ 
cative  and  since  such  factors  cannot  reasonably  be  stated  to  a 
greater  accuracy  than  about  +  0.05,  it  is  just  as  important  as 
with  other  methods  that  the  inputs  result  from  a  searching  probe 
of  highly  experienced  technical  experts.  In  other  words,  the 
apparent  simplicity  of  the  method  has  not  relaxed  the  demand  that 
high  quality  personnel  take  key  roles  in  the  analysis.  On  the 
other  hand,  once  a  baseline  cost  estimate  has  been  formulated  by 
engineering  cost  estimating  methods,  the  analysts  should  be  able 
to  prepare  a  cost  estimate  using  risk  factors  in  a  relatively 
short  time.  The  length  of  time  will  depend  on  the  difficulty  an 
analyst  has  in  consulting  his  technical  experts  and  on  the  level 
of  breakdown  of  his  WBS. 

2 .  Input/Output 

a.  Input 

The  primary,  and  generally  pre-existing,  "input"  of  a  risv 
factor  assessment  is  the  WBS  or  cost  breakdown  of  a  baseline  cost 
estimate.  The  risk  factors  are  formulated  intuitively  based  upon 
analyst  or  subject  matter  expert  experience  and  knowledge  of 
program  hazards.  More  detailed  discussion  of  the  thinking  that 
might  underlie  risk  factor  determination  can  be  found  in  111]. 


b.  Output 

The  "output"  of  a  risk  factor  application  is  a  budget  or 
cost  estimate  increased  over  the  baseline  budget  (or  estimate)  by 
an  amount  anticipated  to  be  sufficient  to  accommodate  otherwise 
indeterminate,  but  probable,  program  costs. 
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3.  Resource  Requirements 

Resource  requirements  for  this  method  can  be  quite  flexible. 
Frequently  the  same  cost  estimator  responsible  for  the  baseline 
estimate  provides  the  risk  factor  result  in  a  few  hours.  It  is 
assumed  that  his  experience,  coupled  with  his  questioning  of  sub¬ 
ject  matter  experts  during  formulation  of  the  baseline  estimate, 
is  satisfactory. 

4.  Application 

Risk  factor  methods  are  the  most  widely  used  of  those  sup¬ 
porting  U.S.  Army  TRACE  procedures  (see  Appendix  I),  and  analysts 
practiced  in  carrying  them  out  can  be  found  in  the  cost  analysis 
offices  of  any  major  Army  development  command. 
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CHAPTER  V 


A.  GENERAL 


TECHNIQUE  SELECTION 


This  chapter  describes  the  circumstances  influencing  the 
selection  of  risk  assessment  methods  and  provides  guidance  which, 
while  not  making  selection  automatic,  clarifies  advantages  and 
disadvantages  of  specific  choices. 

Section  B  discusses  individual  technique  features  for  com¬ 
parative  purposes. 

Section  C  briefly  discusses  conditions  affecting  the  PMO 
which  influence  technique  selection. 

Section  D  discusses  the  purposes  for  which  quantitative  risk 
assessment  can  be  used  and  shows  that  although  these  purposes  are 
partially  program-phase  dependent,  risk-laden  decisions  must  be 
made  throughout  a  program's  life  cycle.  The  section  briefly 
describes  the  extent  to  which  each  technique  supports  the  pur¬ 
poses  listed. 

Section  E  provides  a  table  which  integrates  the  discussions 
of  the  previous  sections  by  characterizing  each  technique  in 
terms  of  its  requirements,  its  applicability,  its  theoretical 
soundness,  and  the  amount  of  insight  it  provides. 

B.  TECHNIQUE  FEATURES 

This  section  summarizes  material  that,  while  partially 
discussed  previously,  should  be  clarified  for  comparative  pur¬ 
poses.  In  this  section  the  different  features  of  specific  tech¬ 
niques  are  identified  and  discussed.  This  discussion  assists  in 
later  sections  which  describe  how  the  conditions  surrounding  a 
program  and  the  purposes  for  which  a  risk  assessment  is  performed 
affect  the  desirability  of  one  technique  relative  to  another. 
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Features  to  be  considered  includes 

o  Ease  of  use 

o  Soundness  of  theory 

o  Duration  of  support  requirement 

o  Versatility 

o  Decision  level  suitability 
o  PMO  committment 

o  Output  content 

The  following  paragraphs  contrast  the  techniques  by  the  cited 
features . 

1 .  Ease  of  Use 

Ease  of  use  comprises  an  aggregate  (subjective)  evaluation 
consisting  of  s  (1)  theoretical  difficulty,  (2)  volume, 
complexity,  and  difficulty  of  determining  inputs,  and  (3)  availa¬ 
bility  of  software  or  other  support.  The  most  difficult  tech¬ 
nique  to  use,  if  only  for  the  manpower  required,  is  probably  an 
(advanced)  network  assessment  being  applied  for  program  control. 
Continual  reporting  is  required  and  manpower  must  be  permanently 
assigned.  The  next  most  difficult  technique  is  the  fully  applied 
type  of  decision  analysis  exemplified  in  [3],  Decision  analysis 
is  of  only  moderate  theoretical  difficulty,  but  both  subject 
matter  and  analytical  insights  are  needed  for  establishing  cre¬ 
dible  inputs  ani  insuring  full  identification  of  problems  and 
resolutions.  The  seemingly  simple  probabilities  associated  with 
the  inputs  are  easily  misreported,  and  sophisticated  interview 
techniques  are  required.  Of  nearly  equal  difficulty  is  the  stan¬ 
dard  network  analysis.  Its  difficulty  stems  from  the  massiveness 
of  its  input  and  the  difficulty  inherent  in  program  network 
modeling.  Perhaps  easier  to  use  are  the  method-of -moments  and 
WBS  simulation  methods.  Sophisticated  interview  techniques  are 
still  necessary,  but  data  input  volume  can  be  held  to  a  more 
managable  level  (no  more  that  40  to  60  cost  elements  vs  200  to 


1000  activities  in  a  network  and  100  or  more  problems  in  a  deci¬ 
sion  analysis).  The  difficulty  in  understanding  the  theory  of 
the  method-of -moments,  while  not  an  obstacle  to  computation, 
makes  it  harder  to  use.  Since  software  for  support  of  methoa-of- 
moments  analyses  is  not  Government  owned,  the  method  is  not  as 
immediately  available  as  those  supported  by  Government-owned 
software.  The  last  three  methods — graphics,  estimating  rela¬ 
tionships,  and  risk  factors — are  all  fairly  simple  in  concept  and 
application:  the  graphic  technique,  because  of  its  possible  data 
acquisition  difficulty,  is  judged  to  be  the  most  difficult. 

2 .  Soundness  of  Theory 

Soundness  of  theory  here  covers  both  basic  theoretical  vali¬ 
dity  and  the  necessity  for  weakening  assumptions.  All  techniques 
require  some  degree  of  simplifying,  or  weakening  assumptions. 
For  networks,  assumptions  are  made  that  the  selected  levels  of 
detail  are  indeed  adequate  and,  usually,  that  inputs  are  sta¬ 
tistically  independent.  Decision  analytic  methods  have  similar 
types  of  assumptions,  and  these  two  methods  are  probably  the  most 
theoretically  valid,  followed  by  WBS  simulation.  The  latter  may 
be  considered  to  be  a  weaker  method  in  that  less  detail  is 
included  and  the  effects  of  all  risks  on  cost  must  be  subjec¬ 
tively  assessed.  In  the  method  of  moments,  it  is  necessary  to 
assume  a  PDF  type  for  total  cost,  which  further  weakens  this 
technique.  The  use  of  the  discontinuous  halves  of  normal  PDF's 
in  the  graphic  method  causes  it  to  be  judged  even  weaker. 
Because  the  estimating  relationship  method  and  the  risk  factor 
method  both  require  substantial  assumptions  of  comprehensive 
natures,  the  soundness  of  the  theory  for  these  methods  is  judged 
to  be  low,  although  t-hey  have  gained  widespread  acceptance. 

3 .  Duration  of  Support  Requirement 

Duration  of  support  requirement  varies  from  perpetual  (for 
the  advanced  network  assessment  systems  used  for  program  control) 
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to  hours  (for  risk  factors  and  estimating  relationships).  Table 
1  in  Section  E  gives  comparative  range  estimates. 

4 .  Versatility 

A  more  versatile  ♦‘echnique  can  be  used  for  a  larger  number  of 
applications  than  a  less  versatile  one.  Applications  are  shown 
in  Table  1. 


5 .  Decision  Level  Suitability 

The  type  of  output  produced  by  some  methods  can  be  considered 
of  more  or  less  value  to  any  of  three  decision  levels,  i.e., 
within  a  PMO,  at  the  program  manager  level,  and  at  levels  above 
the  program  manager.  For  levels  within  a  PMO,  program  element 
managers  will  need  assessments  producing  significantly  detailed 
output  descriptive  of  subelement  risks.  A  program  manager  may  be 
best  served  by  more  aggregated  outputs  descriptive  of  the  major 
program  elements  but  capable  of  describing  subelement  risks  on  an 
as-needed  basis.  Levels  above  the  program  manager  will  probably 
want  more  aggregated  information  providing  insights  into  program 
general  status  and  risks  of  major  alternatives.  See  Table  1. 

6 .  PMO  Committment 

For  this  handbook's  purpose,  PMO  Committment  represents  the 
amount  of  attention  required  from  a  PM  or  his  first  level  subor¬ 
dinates  for  a  high  quality  assessment.  The  manhours  shown  in 
Table  ]  are  subjective  estimates.  For  the  network  and  the  deci¬ 
sion  analysis  assessments,  the  estimates  are  based  on  the  PM's 
devoting  approximately  two  hours  of  briefing  and  guidance  time 
per  week  to  the  assessment  team.  For  the  method-of -moments , 
Probabilistic  Event  Analysis  (PEA),  WBS  simulation  estimating 
relationship,  and  risk  factor  methods,  the  reduced  scope  of  the 
assessment  is  assumed  to  permit  sufficient  guidance  on  underlying 
program  model  structure  (e.g.,  WBS)  and  assurance  of  reliability 
of  data  to  require  a  small  amount  of  the  PM's  time.  Should  the 
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method-of -moments,  graphics,  or  risk  factor  methods  be  executed 
by  the  PM  himself  or  his  first  level  subordinates,  of  course  more 
personal  committment  will  be  required.  Should  a  graphics 
assessment  be  accomplished  by  a  supporting  analyst,  the  PM  can 
probably  reduce  his  committment  to  be  commensurate  with  the  rough 
approximation  nature  of  such  an  assessment. 

7.  Output  Content 

A  technique  with  a  greater  output  content  provides  more 
breadth,  depth,  and  understanding  than  one  with  outputs  of  a 
lesser  content.  While  output  content  is  associated  with  output 
quantity,  it  differs  in  that  significant  by-product  benefit  may 
accompany  some  assessment  processes  and  not  others. 

Network  techniques  provide  the  direct  benefits  of  insights 
into  program  element  costs  and  risks  resulting  from  developing 
activity  PDF's  at  a  selected  level  of  detail.  Additionally, 
assessments  are  inherently  available  for  selected  subsidiary 
parts  of  programs.  The  advanced  network  systems  supplement  such 
snapshot  capability  by  providing  a  continuing  status  reporting 
and  projective  capabiltiy.  The  significant  by-product  of  the 
detailed  model -building  process  for  a  network  assessment  is  the 
improvement  in  program  definition  (sometimes  unrecognized)  that 
results.  The  decision  analysis  methods  inherently  sacrifice 
detail  for  clarity.  While  the  full  scale  assessment  can  be  a 
superb  analytical  tool,  it  cannot  provide  to  the  PM  or  higher 
levels  the  breadth  of  a  detailed  network  model.  Its  capability 
for  showing  tradeoff  consequences  cannot  be  approached  by  other 
techniques,  however.  Roughly  the  same  output  types  result  from 
method-of -moments,  WBS  simulation  and  graphic  assessments  in  that 
some  level  of  PDF  is  provided  for  each  cost  element.  A  wel] 
carried  out  Probabilistic  Event  Analysis  provides  probabilistic 
statements  relative  to  problems  that  can  approach  the  same  level 
of  detail.  Estimating  relationship  and  risk  factor  methods 
aggregate  information  and  contain  such  structural  weakness  as  to 
cause  them  to  be  judged  lower  in  output  content. 
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C.  CONDITIONS  AFFECTING  THE  PMO 


1.  General 

While  it  might  seem  that  comparing  the  characteristics  of  a 
required  risky  decision  with  the  attributes  of  available 
assessment  techniques  might  suffice  for  technique  selection,  a 
number  of  conditions  external  and  internal  to  the  PMO  can  modify 
technique  desirability.  Such  conditions  include  high  level  poli¬ 
cies,  the  military  service  and  its  policies,  available  program 
support  organizations,  management  level  of  the  PMO,  resource 
availability,  program  phase,  program  schedule  and  technical 
complexity,  and  program  criticality  (both  budgetary  and 
military) . 

In  general  those  programs  that  are  of  high  criticality,  high 
management  level,  well  funded,  complex,  or  any  combination  of 
these  factors,  can  obtain  any  of  the  more  demanding  and  rewarding 
techniques.  Small  programs  headed  by  a  "project  engineer", 
"program  coordinator",  or  "product  manager",  (for  whom  Table  1(b) 
is  applicable)  however,  may  be  constrained  by  their  own  resources 
or  the  priorities  of  support  organizations  to  use  only  the 
speedier  and  less  detailed  techniques.  Even  a  well-funded,  self 
contained  program  office  may  at  times  have  subsidiary  programs  or 
time  constraints  which  may  make  these  speedier  techniques 
appropriate. 

The  amount  and  depth  of  quantitative  risk  assessment  obtained 
by  self  contained  PMOs  can  usually  be  determined  by  the  program 
manager.  He  is  likely  to  need  to  call  on  the  host  command's 
systems  analysis  office  for  other  than  POM/Budget  (TRACE)  risk 
assessment,  and  he  will  need  to  train  members  of  his  staff  or 
obtain  contract  services  for  ongoing  risk  assessment,  such  as  for 
project  control.  For  assessments  of  a  periodic  nature  he  can,  at 
least  in  the  Army,  find  experienced  support  offices,  or  he  can 
obtain  contract  services. 


The  program  manager  of  a  matrix-supported  PMO  can  usually 
determine  the  timing  and  depth  of  risk  analysis  provided  by  the 
host  command  (within  the  latter's  priorities),  but  may  have  to 
obtain  contract  services  if  the  systems  analysis  support  office 
is  overloaded.  Continuing  risk  assessment  most  likely  must  come 
from  a  contractor.  If  a  matrix  support  office  is  used,  very 
often  it  may  only  be  familiar  with  one  method  and  will  prefer  to 
use  that  particular  method. 

Experience  shows  that  the  "project  engineer/program 
coordinator/product  manager"  is  unlikely  to  obtain  support  for 
risk  assessment,  and  he  often  possesses  insufficient  funds  to 
contract  for  it.  The  energetic  and  innovative  PC  can,  however, 
utilize  graphic,  decision  analytic,  and/or  method-of -moments 
techniques  to  obtain  a  better  definition  of  the  cost  and  schedule 
risks  of  his  program. 

2.  Policy  Requirements 

It  might  be  expected  that  higher  level  and  military  service 
policies  would  help  in  the  selection  of  a  risk  assessment  tech¬ 
nique,  and  this  is  so  to  a  limited  degree.  A  listing  of  sum¬ 
marized  and  annotated  higher  and  service  level  policy  directives 
appears  in  Appendix  H.  For  purposes  of  this  chapter,  the 
following  overview  suffices. 

The  higher  level  policies  relating  to  risk  assessment  derive 
from  OMB  Circular  A-109,  DOD  Directive  5000.1,  and  DOD  Instruc¬ 
tions  5000.2,  5000.38,  and  7041.3.  Only  in  the  last  is  there  any 
reference  to  quantitative  risk  assessment,  and  that  has  an  infor¬ 
mative  rather  than  directive  sense.  On  the  other  hand,  the  first 
four  do  make  statements  generally  supporting  quantitative  risk 
assessment  concepts.  The  extensive  list  of  Army  directives 
relating  to  quantitative  risk  assessment  can  be  found  in  Appendix 
H.  In  summary,  they  specify  chat  major  and  In  process  Review 
(IPR)  programs  will  supply  Decision  Risk  Analyses  (DRA's)  for 
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each  milestone  review.  While  this  does  not  explicitly  specify 
the  assessment  techniques  to  be  used,  network  analysis  has  become 
the  customary  technique  employed  for  the  DRA,  therefore  any  tech¬ 
nique  selected  for  programs  of  similar  criticality  would  probably 
be  expected  to  be  of  similar  quality. 

The  second  explicit  Army  policy  relating  to  a  type  of  risk 
assessment  stems  from  the  TRACE  budget  procedure.  The  example 
given  in  the  TRACE  Letter  of  Instruction  (LOI)  [4]  uses  the  risk  fac¬ 
tor  method  but  an  early  paragrph  permits  use  of  any  generally 
accepted  technique.  As  with  DOD  directives,  other  Army  direc¬ 
tives  are  compatible  with  quantitative  risk  assessment  concepts 
but  do  not  require  any  particular  techniques. 

Of  the  USAF  directives  discussing  risk  assessment  or  related 
subjects,  only  AFR  173-11,  paragraph  7.e.,  under  Independent  Cost 
Analysis  (ICA)  Methodology  contains  any  directive  sense  for  risk 
assessment.  While  its  prescription  that  the  ICA  will  use  fre¬ 
quency  distributions  for  cost  ranges  precludes  employment  of  risk 
factors  and  estimating  relationships,  any  of  the  other  methods 
shown  in  this  handbook  meet  the  requirement. 

The  Naval  Material  Command  has  decided  to  evaluate  the  possi¬ 
bility  of  adapting  the  Army  TRACE  procedure  to  Navy  use.  To 
investigate  this  possibility,  the  Naval  Air  Systems  Command  has 
performed  a  few  preliminary  cost  estimates  using  one  or  another 
of  the  procedures  associated  with  TRACE  and  has  published  NAVAIP 
Instruction  7131.1,  dated  21  April  1983;  Subject:  Management  of 
Research,  Development,  Test,  and  Evaluation,  Navy  (RDT&E,N)  Risk 
Cost  Estimate  Funding. 

This  instruction,  however,  prescribes  only  the  procedures  for 
holding,  allocating,  or  reprogramming  funding  requested  for  the 
purpose  of  covering  identified  program  uncertainties.  It  does 
not  address  methods  for  determining  the  amount  to  be  requested. 
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Although  the  Naval  Sea  Systems  Command  has  successfully 
employed  at  least  two  advanced  network  assessment  systems  for 
management  control  in  a  number  of  important  programs,  no  direc¬ 
tives  on  the  subject  of  quantitative  risk  assessment  were  found 
during  handbook  preparation.  Likewise,  no  NAVELEX  directives 
were  discovered. 

The  Headquarters,  Naval  Material  Command,  has  published  the 
Navy  Program  Manager's  Guide,  dated  April  1983,  which  includes 
sections  on  Risk  Management  (page  3-21)  and  Budgeting  for  Program 
Risk .  These  sections  recommend  risk  assessments  be  included  in 
program  acquisition  strategies,  but  they  do  not  specify  any  tech¬ 
nique  to  be  used. 

D .  Applications 

Quantitative  risk  assessments  have  been  proposed  for 
assisting  in  at  least  the  following  decision  categories: 

o  Program  control 
o  Major  planning  decisions 
o  Acquisition  strategy  selection 
o  Technical  alternative  selection 
o  Contract  structuring 
o  Source  selection 
o  POM/Budget  establishment 

While  for  some  applications  no  actual  use  of  the  application 
has  been  discovered  during  handbook  preparation,  there  does  not 
appear  to  be  any  obstacle  preventing  their  use.  All  of  these 
decision  categories  represent  aspects  of  program  management  in 
which  decisions  must  be  made  under  conditions  of  uncertainty. 

1 .  Program  Control 


"Catch  phrases"  in  management  can  become  a  burden  to  the 
program  manager.  Some  program  managers  have  felt  that  risk 


assessment  has  been  used  in  a  "catch  phrase"  way,  and  that  the 
assessment  they  were  required  to  undertake  did  not  provide  them 
with  anything  useful.  Their  expressed  attitude  was,  "The 
assessment  described  a  changing  program  in  static  terms.  By  the 
time  the  analysts  were  finished,  they  were  analyzing  a  program 
that  no  longer  existed.  What  I  need  is  an  analysis  that  responds 
to  the  changes." 

Such  statements  lead  to  the  important  decision  category 
Program  Control.  Program  control  means  the  monitoring  of  plans, 
schedules,  and  costs  to  ensure  that  standards  are  met  and  to 
allow  for  timely  corrective  action.  The  Cost  Performance  Report 
and  Cost  Schedule  Status  Reports  made  by  contractors  under  the 
Selected  Acquisition  Information  Management  System  and  conforming 
to  Cost/Schedule  Control  Systems  Criteria  were  designed  to  pro¬ 
vide  contractors  and  program  managers  with  information  for  mana¬ 
gement  control.  As  valuable  as  such  reports  are,  they  leave  it 
to  reviewers  to  draw  broad  intuitive  conclusions.  Some  risk 
assessment  systems  (see  Appendix  F  and  Reference  [1]  )  have  been 
implemented  and  used  for  management  control  for  extended  periods 
in  the  development  of  major  weapons  systems.  Although  it  is 
impossible  to  measure  the  difference  in  the  management  perfor¬ 
mance  c  that  might  have  resulted  without  the  systems,  it  can  be 
said  that  the  program  managers  were  satisfied  with  the  systems. 

2 .  Planning 


Planning  overlaps  some  of  the  other  decision  categories 
(e.g .,  acquisition  strategy  selection),  and  it  is  perhaps  the 
most  pervasive  of  decision  activities  since  planning  is  required 
during  all  program  phases.  For  major  planning  decisions — those 
in  which  important  program  direction  is  determined — a  program 
manager  may  be  willing  to  invest  significant  resources  and  per¬ 
sonal  attention  into  understanding  whet  his  community  of  experts 
collectively  believe  are  the  chances  of  success  of  each  alter¬ 
native  plan. 
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3.  Acquisition  Strategy  Selection 

Acquisition  strategy  selection  is  a  category  of  early  program 
decision  in  which  risk  assessment  is  likely  to  provide  support  in 
a  somewhat  indirect  way.  Certain  aspects  of  acquisition  strategy 
selection,  such  as  the  decision  to  undertake  parallel  R&D  by  com¬ 
petitive  contractors  rather  than  by  selection  of  a  single  com¬ 
petitor,  can  be  analyzed  explicitly.  On  the  other  hand,  the  risk 
effects  of  selecting  particular  contract  types  may  not  be  amen¬ 
dable  to  quantified  analysis.  In  this  case  it  would  be  premature 
to  assert  that  a  quantified  risk  assessment  would  do  more  than 
indicate  the  need  or  lack  of  need  for  a  specific  type  of 
contract.  See  paragraph  5,  on  the  following  page,  for  discussion 
of  how  certain  types  of  contracts  can  be  structured  using  risk 
assessment. 

4.  Technical  Alternative  Selection 


The  next  decision  category  amenable  to  quantitative  risk  ana¬ 
lysis  is  technical  alternative  selection.  Technical  alternative 
selection  must  be  undertaken  soon  after  a  mission  need  is  iden¬ 
tified,  and  it  is  during  such  concept  formulation  stages  that 
uncertainty  is  greatest  and  risk  assessment  most  needed.  A 
program  frequently  passes  through  this  phase  before  it  is 
designated  as  a  program  or  has  a  program  manager.  On  the  other 
hand,  as  a  program  passes  through  each  subsequent  phase,  the 
manager  must  go  through  progressively  narrower  selections  of 
technical  alternatives,  ranging  from  major  weapon  system  tech¬ 
nology  alternatives  to  choices  of  components  and  materials.  Each 
alternative  represents  a  collection  of  large  or  small  uncertain¬ 
ties  regarding  cost,  schedule,  technical  performance,  and  program 
plan.  In  any  event,  there  will  be  a  community  of  experts  on  whom 
the  manager  will  call  for  opinions  on  each  aspect  of  the  uncer¬ 
tainties  associated  with  the  alternatives.  The  manager  will  want 
an  understanding  of  how  the  uncertainties  relate  tc  one  another 
and  how  the  alternatives  compare. 
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5.  Contract  Structuring 


Once  an  acquisition  strategy  has  been  selected  that  includes 
incentive  type  contracts,  program  management  is  faced  with  the 
decision  of  incentive  fee  determination.  Since  the  purpose  of 
incentive  contracts  is  the  sharing  of  risk  between  the  contractor 
and  the  Government  and  the  provision  of  an  incentive  to  the 
contractor  to  control  cost  risk,  an  assessment  of  risk  is  needed. 
Appendix  K  describes  a  way  for  accomplishing  this  with  any  of  the 
quantitative  risk  assessments  producing  a  CDF. 

6.  Source  Selection 


Source  selection  often  does  not  involve  the  program  manager 
directly,  yet  source  selection  evaluation  boards  are  required  to 
evaluate  risks  as  a  determinant  of  selection.  A  quantified  eva- 
luation  provides  capability  to  substantiate  the  risk  evaluation 
with  improved  credibility.  Time  constraints  on  a  source  selec¬ 
tion  process  limit  the  detail  of  such  an  assessment  and  imply  the 
desirability  of  using  one  of  the  less  demanding  methods.  The 
earliest  and  latest  program  phases  may  seem  to  involve  source 
selections  of  a  less  risky  nature  than  does  the  development 
phase,  but  the  deployment  phase,  for  example,  may  require  modifi¬ 
cation  program  source  selections  of  a  highly  uncertain  character. 

7 .  Budget  Formulation 

Budget  formulation  is  a  decision  category  required  on  a 
periodic  basis.  The  decision  as  to  what  level  of  budget  to 
request  can  be  made  on  the  basis  of  a  quantified  evaluation  of 
the  risk  of  failing  to  achieve  objectives  with  a  given  funding 
level.  Implicit  is  the  growing  need  for  quantified  risk 
assessment  to  support  credible  budget  requests. 


E.  TECHNIQUE  SELECTION  CRITERIA 

Tables  1(a)  and  (b)  summarize  and  integrate  the  subjects 
discussed  in  previous  sections  so  that  a  best  fit  of  technique  to 


situation  may  be  achieved.  Table  1(a)  provides  criteria  for  a 
self  contained  or  a  designated,  chartered,  and  matrix  supported 
PMO,  assuming  the  assessment  is  to  address  issues  appropriate  to 
the  PM's  attention.  Table  1(b)  provides  criteria  for  a  program 
coordinator,  project  engineer,  or  product  manager  (herein 
designated  PC)  to  address  of  issues  appropriate  to  him  or  to  a 
time  or  resource  constrained  PM  of  a  higher  level  program. 


TABLE  1  (b) 

TECHNIQUE  SELECTION  CRITERIA 
FOR 

LESS-THAN-MAJOR  PROGRAMS 
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CHAPTER  VI 


IMPLEMENTATION 

A.  General 

The  program  manager  who  wishes  to  obtain  a  quantitative  risk 
assessement  can  obtain  it  from  within  his  own  PMO,  through  a 
functional  support  office  of  his  host  command,  from  his  prime 
contractor,  or  from  a  support  services  contractor.  Each  source 
has  been  found  appropriate  and  has  been  used  by  at  least  one 
program  manager,  but  it  was  determined  during  preparation  of  this 
handbook  that  most  P 40  personnel  preferred  to  use  functional  sup¬ 
port  offices.  Some  criteria  considered  when  selecting  a  source 
for  accomplishing  a  quantitative  risk  assessment  include:  (1) 

the  need  for  objectivity,  f2)  difficulty  of  keeping  less  closely 
coupled  organizations  current  on  dynamic  programs,  and  (3)  the 
requirement  for  familiarity  with  technical  and  programmatic 
problems,  practices,  and  polices.  Some  program  managers  find 
standardization  desirable,  believing  that  risk  assessments  may  be 
better  understood  by  reviewing  authorities  and  can  be  used  to 
compare  programs.  Table  2  summarizes  the  application  of  these 
criteria  to  the  sources  of  risk  assessment  support.  In  Table  2 
Capability  refers  to  the  knowledge  of  risk  assessment  techniques, 
and  Objectivity  is  self  explanatory.  Standardization  means  the 
use  of  a  locally  standard  technique  and  of  uniformly  applied  cri¬ 
teria.  Responsiveness  and  Familiarity  aggregate  the  ideas  of 
technical  and  program  knowledge  with  adaptability  to  program 
changes . 
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TABLE  2 


CRITERIA 


Source 

Capability 

Objectivity 

Standardization 

Responsiveness 
and  Familiarity 

PMO 

occasional , 
Limited 

purpose 

dependent 

enforceable 

high 

MTRX 

service 

specific 

service 

specific 

service 

specific 

variable 

PRIME 

variable 

questionable 

enforceable 

high 

SSC 

available 

purpose 

dependent 

enforceable 

available 

The  entries  in  Table  2  deserve  some  expansion.  The  term 
"service  specific"  indicates  that  those  qualities  vary  by  service 
as  a  result  of  inherent  organizational  relationships. 

The  entry  for  PMO  capability,  "occasional,  limited,"  recogni¬ 
zes  that  some  PMOs  contain  members  who  have  developed  the  interest 
and  skills  to  perform  risk  assessment.  The  burden  of  other 
duties  usually  has  limited  the  scope  of  assessments  to  the  more 
summary  types.  On  the  other  hand,  in  the  Naval  Sea  Systems 
Command,  support  services  contractors  have  developed  systems  to 
assist  the  PMO  in  its  continuing  management  of  project  risk.  In 
one  case  the  systems  were  operated  by  full  time  PMO  members  in  a 
program  of  the  type  having  to  depend  on  matrix  organizations  for 
much  of  its  daily  function.  The  program  was,  however,  one  of 
sucn  high  visibility  and  cost  as  to  justify  the  effort  and  expen¬ 
diture  required  for  this  application. 

The  entry  for  Prime  Capability,  "variable,"  recognizes  that 
some  prime  contractors  may  not  have  developed  a  significant  risk 
assessment  capability. 

In  the  objectivity  column,  the  entries  "purpose-dependent" 
appear  because  neither  a  member  of  the  PMO  nor  a  support  services 
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contractor  are  likely  to  provide  results  that  are  any  more  objec¬ 
tive  than  are  desired  by  th^  controlling  manager. 


The  entry  "enforceable"  under  Standardization  indicates  that 
orders  to  PMO  members  and  contracts  to  contractors  can  require 
the  use  of  a  specific  type  of  technique.  Unless  the  require¬ 
ments  are  specified,  it  can  be  expected  that  the  resulting  risk 
assessments  from  these  sources  will  not  be  standardized.  Risk 
assessments  done  by  the  functional  support  groups  will  probably 
be  accomplished  in  a  form  and  format  standardized  within  that 
command . 

In  the  final  analysis,  the  first  source  from  which  a  program 
manager  should  seek  a  comprehensive  risk  assessment  is  the 
Government  (host  command)  matrix  organization.  Should  that 
source  be  unable  to  provide  it,  and  if  the  capability  or  resour¬ 
ces  are  not  available  within  the  PMO,  then  because  of  capabiltiy 
and  objectivity,  probably  the  next  best  alternative  is  a  support 
services  contractor.  Summary  level  assessment  can  be  accomplish¬ 
ed  by  PMO  technical  personnel  if  care  is  taken  with  gathering 
information. 

A  more  complete  list  of  service,  commercial,  and  academic 
centers  of  research  and  expertise  is  included  as  Appendix  J. 

B.  Service  Considerations 


i.  All  Services 


For  information  of  a  general  nature  or  which  may  cross 
service  lines,  contact  may  be  made  with: 


o  The  Research  Directorate  (DRI-R) 
Defense  Systems  Management  College 
Fort  Belvoir,  VA  22060, 

(703)  664-5783 
AV  354-5783 


2.  Army 


In  general.  Army  materiel  development  commands  provide  risk 
assessment  support  to  resident  PMO's  through  one  or  two  analysis 
organizations.  The  names  vary,  but  all  are  given  titles  that 
closely  approximate  "Co3t  and  Systems  Analysis."  Sometimes  the 
two  functions  are  separated  and  sometimes  not,  but  in  general,  a 
systems  analysis  unit  performs  Decision  Risk  Analyses  (DRA's)  and 
risk  assessments  of  a  programmatic  nature,  and  a  cost  analysis 
unit  performs  Total  Risk  Assessing  Cost  Estimate  (TRACE)  related 
assessments  for  budgetary  purposes. 

Each  command  tends  to  adopt  and  become  most  proficient  in  the 
use  of  particular  techniques.  For  example,  most  systems  analysis 
offices  have  adopted  network  analysis  using  the  VERT  computer 
model  as  the  vehicle  for  conducting  DRAs,  while  MICOM  has  deve¬ 
loped  a  capability  to  employ  network  analysis  using  the  RISNET 
model.  Most  of  the  cost  analysis  offfices  make  use  of  the  method 
of  risk  factors  to  formulate  TRACE'S. 

Should  a  program  manager  wish  to  undertake  a  more  sophisti¬ 
cated  and  detailed  risk  asessment  or  risk  analysis  than  a  command 
systems  analysis  office  is  prepared  to  provide,  the  latter  can 
identify  support  services  contractors  capable  of  such  analyses, 
as  most  systems  analysis  offices  have  employed  contractors  to 
assist  them  during  overload  situations. 

For  additional  information  on  Army  policy  and  practices  or 
for  other  assistance,  Department  of  the  Army  personnel  may 
contact: 

o  HQ  DARCOM 

DRCCP-ER  (Mr.  Richard  Baker) 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333 
(202)  274-8030 


o 


U.S.  Army  Material  System  Analysis  Activity  Army  Pro¬ 
curement  Research  Office 
Fort  Lee,  VA  23801 
(804)  734-2027 
AV  687-2027 


3 .  Air  Force 

The  comptroller's  office  at  Armament  Division  maintains  a 
network  analysis  computer  program  referred  to  as  "the  RISK  model". 


At  many  Air  Force  Systems  Command  locations,  risk  assessment 
is  the  interest  primarily  of  the  various  controller  offices,  each 
of  which  develops  estimates  for  engineering  change 
order/management  reserves. 


Points  of  contact  for  Department  of  the  Air  Force  personnel 
needing  additional  information  on  Air  Force  and  local  policies/ 
procedures  are  the  following: 


o  HQ  AFSC/ACCE 

Andrews  Air  Force  Base,  D.C.  20334 
(301)  981-4306 
AV  858-4306 


o  ASD/ACCR 

Wright-Patterson,  AFB ,  OH  45433 
(513)  255-5904 
AV  785-5904 


o  ESD/ACC 

Hanscom  AFB,  MA  01731 
(617)  861-2677 
AV  478-2677 

o  AD /ACC 

Eglin  AFB,  FL  32542 
(904)  882-2151 
AV  872-2151 


o  Air  Force  Business  Management  Research  Center  (BMRC) 
Wright-Patterson  AFB,  OH  45433 
(513)  255-6221 
AV  785-6221 
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The  functional  support  groups  of  Navy  Systems  Commands  have 
relatively  little  experience  in  conducting  formal  risk 
assessment.  A  survey  supporting  preparation  of  this  handbook 
found  that  the  Navy  has  generally  depended  on  support  services 
contractors  to  conduct  risk  assessments. 


For  personnel  of  the  Department  of  the  Navy  more  information 
can  be  obtained  by  contacting  the  following: 

o  Executive  Director 

Navy  Office  for  Acquisition  Research 
(Located  at  DSMC,  Fort  Belvoir,  VA  22060) 

(703)  664-2817 
AV  354-2817 

o  Na  ’al  Sea  Systems  Command 

SEA-001 

Washington,  D.C.  20362 
(202)  692-8600 
AV  222-8600 

o  Naval  Air  Systems  Command 

AIR-104A 

Washington,  D.C.  20361 
(202)  692-0066 
AV  222-0066 
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APPENDIX  A 
BIBLIOGRAPHY 


This  bibliography  of  the  risk  analysis  literature  is  divided 
into  five  categories:  basic  discussion  of  general  risk  assess¬ 
ment  procedures,  theoretical  discussion  of  mathematical  tech¬ 
niques,  case  studies,  administrative  procedures  for  risk  manage¬ 
ment  or  budget  procedures,  and  other  topics. 

A  matrix  format  for  each  category  is  included  for  easy  access 
to  the  literature.  In  the  matrix,  literature  is  coded  for  the 
following  subjects:  acquisition  phase  specifically  mentioned  or 
most  appropriate,  management  area  (budgeting,  planning,  schedul¬ 
ing,  controlling,  pricing,  and  strategy  selection),  risk  varia¬ 
ble  ( s ) ,  and  analytical  approach  discussed  above  in  the  text. 

Each  work  is  numbered  in  each  appropriate  matrix,  and  the 
full  bibliographic  information  for  that  numbered  work  is  provided 
in  the  accompanying  alphabetical  list.  The  full  bibliographic 
information  includes,  where  it  is  known,  AD/LD  number  for  obtain¬ 
ing  publications  through  the  Defense  Technical  Information  Center 
or  the  Defense  Logistic  Information  Exchange,  and  annotations  as 
well  as  the  standard  publication  information.  Asterisks  appear 
beside  entries  (core  items)  providing  information  of  key  impor¬ 
tance  to  understanding  a  technique  or  serving  as  a  superior 
example  of  application. 


BASIC  DISCUSSION 


*1.  Antunes,  Moore,  et  al,  Ed.,  Army  Programs  Decision  Risk 
Analysis  (DRA)  Handbook,  U.S.  Army  Logistics  Management  Center, 
Fort  Lee,  VA,  (DARCOM  Handbook  11-1.1-79),  1979,  (#). 

Subjective  probability  methods  discussed:  Choice  between 
gambles  pdf/cdf,  standard  lottery,  Churchman/Ackof f ,  Delphi. 
Describes  role  of  DRA  in  material  acquisition  process,  as  well 
as  methods  of  performing  DRA's. 

2.  Atzinger,  E.,  et  al,  Compendium  on  Risk  Analysis  Techniques, 
DARCOM  Materiel  Systems  Analysis  Activity,  Aberdeen  Proving 
Ground,  MD,  1972,  (AD  746  245),  (LD  28463). 

Subjective  probability  methods  discussed:  Choice  between 

gambles  pdf/cdf,  standard  lottery,  Churchman/Ackof f ,  Delphi. 

3.  Babiarz,  A. A.,  CAPT,  USAF,  and  Giedras,  P.W.,  CAPT  USAF,  A 
Model  To  Predict  Final  Cost  Growth  in  a  Weapon  System  Develop¬ 
ment  Program,  The  School  of  Systems  and  Logistics,  Air  Force 
Institute  of  Technology,  Wright-Patterson  AFB ,  OH,  1975,  (AD  A016 
040)  ,  (LD  34803A) . 

Replicates  Glover-Lenz  application  of  Martin  Cost  Model  which 
incorporates  uncertainty  and  cost  analysis  addressing  final 
weapon  development  cost.  Measures  uncertainty  by  flatness  of 
probability  distribution,  defined  as  entropy. 

4.  Banash,  R.C.,  and  Hurta,  D.W.,  "Risk  Analysis  in  Weapons 
Development" ;  Proceedings  of  the  1972  U.S.  Army  Operations 
Research  Symposium  — 'Risk  Analysis',  U.S.  Army  Operations 
Research  Office  -  Durham,  Durham,  NC,  May  1972,  (AD  748  407),  (LD 
27863)  . 

Discusses  organization  of  risk  analysis  team.  Distinguishes 
risk  analysis  and  decision  risk  analysis.  Defines  decision 
risk  analysis  for  U.S.  Army.  Subjective  probability  assessment 
method — Delphi . 

*5.  Barclay,  Brown,  Kelly,  Peterson,  Phillips,  and  Selvidge, 
Handbook  for  Decision  Analysis,  Decisions  and  Designs  Inc., 
McLean,  VA,  1977,  ( #) . 

Provides  introduction  to  basic  concepts  and  operations  of  deci¬ 
sion  analysis.  Includes  Bayesian  concepts. 

6.  Barclay,  D.H.,  MAJ  USA,  The  Project  Manager  and  Systems 
Analysis ,  Defense  Systems  Management  College,  Fort  Belvoir,  VA, 
1974,  (LD  32631A) . 


1/  *  Core  Item 

(#)  No  AD/LD  number  identified 
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7.  Bevelhymer,  H.L.,  CAPT  USAF,  A  Proposed  Methodology  for 
Weapon  System  Development  Risk  Assessment,  School  of  Engineering, 
Air  Force  Institute  of  Technology,  Wright-Patterson  AFB,  OH, 
1973,  (AD  766  885),  (LD  29823). 

Developed  and  tested  methodology  for  obtaining  input  distribu¬ 
tions  from  routine  contractor-supplied  data.  Used  VERT  model 
to  develop  network.  Applied  method  of  moments.  Tested  metho¬ 
dology  on  actual  weapon  system  in  FSD.  Results  inconclusive f 
but  merit  further  application. 

8.  Brown,  E.L.,  An  Application  of  Simulation  Networking  Tech¬ 
niques  in  Operational  Test  Design  and  Evaluation,  Georgia 
Institute  of  Technology,  May  1975,  (AD  A024  204). 

9.  Busse,  D.E.,  MAJ  USAF,  A  Cost  Performance  Forecasting  Model, 
Air  Command  and  Staff  College,  Air  University,  Maxwell  AFB,  AL, 
May  1977,  (AD  B019  568). 

Estimate  at  Completion  (EAC)  Projection.  Uses  power  curve  and 
regression,  plus  Cost  Performance  Report  (CPR)  information. 

10.  Cost  uncertainty/Manaqement  Reserve  Analysis,  Cost  Manage¬ 
ment  and  Technology  Division,  Directorate  of  Cost  Analysis, 
Deputv  for  Comptroller,  Armament  Division,  Eglin  AFB,  FL,  January 
1982,  (#). 

WBS,  subjective  probability  method  discussed:  centralized 

decision  using  expert  advice.  Risk  Model  Documentation. 

11.  Cox,  L.,  and  Bohn,  M. ,  Report  on  the  Development  of  a  Proto¬ 
type  Computerized  Model  and  Data  Base  for  Use  in  Comparing 
Acquisition  Strategies;  The  Analytic  Sciences  Corporation, 
Arlington,  VA  (TR-1375),  January  1981,  (#). 

Interactive  computer  model  which  compares  acquisition  strate¬ 
gies  based  on  weapon  system  oncept  and  program  objectives. 

12.  Crawford,  L.P.,  LCDR  USN,  A  Case  Study  in  Risk  Decision 
Analysis ,  Defense  Systems  Management  College,  Fort  Belvoir,  VA 
1973,  (AD  A046  651),  (LD  32644A) . 

Description  of  fictitious  application  to  production  decision 
for  missile  system,  using  decision  analysis.  Subjective  proba¬ 
bility  assessment  decentralized  to  group  members. 


1/  *  Core  Item 

(#)  No  AD/LD  number  identified 
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13.  Davis,  G.W.,  The  Dilemma  of  Uncertainties  Associated  with 
Cost  Estimating  in  the  Project  Management  Office,  Defense  Systems 
Management  College,  Fort  Belvoir,  VA,  1976,  (AD  A029  274). 


14.  Dienemann,  P.,  Estimating  Cost  Uncertainty  Using  Monte  Carlo 
Techniques,  The  RAND  Corporation,  Santa  Monica,  CA,  January  1966, 
(AD  629  082)  . 

15.  Dodson,  E.N.,  "Risk  Analysis  in  the  Acquisition  of  BMD 
Systems",  Proceedings  of  the  1972  U.S.  Army  Operations  Research 
Symposium — 'Risk  Analysis’,  U.S.  Army  Operations  Research 
Off ice-Durham,  Durham,  NC,  May  1972,  (AD  748  407). 

16.  Donnell,  M.L.,  and  Ulvila,  J.W.,  Decision  Analysis  of 
Advanced  Scout  Helicopter  Candidates;  Decisons  and  Designs,  Inc., 
McLean,  VA,  February  1980,  (AD  081  483). 

Ilustrates  use  of  technical  alternative  selection  using  multi¬ 
attribute  utility  theory  but  no  decision-analysis-explicit  con¬ 
sideration  of  risk. 

17.  Edgar,  J.  D.,  LTC  USAF,  "Controlling  Murphy:  How  to  Budget 

for  Program  Risk,  Concepts  -  The  Journal  of  Defense  Systems 
Acquisition  Management,  Volume  5,  Number  3,  Defense  Systems 
Management  College,  Fort  Belvoir,  VA  1982,  (#). 

18.  Feiler,  A.M.,  The  TRANSIM  V  Manual,  Volume  I,  Introduction 
to  TRANSIM  V,  UCLA  School  of  Engineering  and  Applied  Science, 
Pasadena,  CA,  UCLA-ENG-7848 ,  September  1978,  (AD  A  059  925);  The 
TRANSIM  V  Manual,  Volume  II,  TRANSIM  User's  Manual. UCLA  School  of 
Engineering  and  Applied  Science,  Pasadena,  CA,  UCLA-ENG-7855 , 
September  1978,  (AD  A  060  397). 

19.  Feltus,  E.E.,  Risk  Analysis  in  the  Engineering  Process, 
Industry/Joint  Services  Automatic  Test  Conference  and  Workshop  on 
Advanced  Test  Technology,  San  Diego,  CA,  1978,  (#). 

Broad  treatment  to  stimulate  interest  in  risk  analysis. 

20.  Hayes,  R.A.,  CAPT  USAF,  An  Evaluation  of  a  Bayesian  Approach 
to  Compute  Estimates-at-Completion  for  Weapon  Systems  Programs, 
Air  University,  Air  Force  Institute  of  Technology,  Wright-Patter- 
son  AFB ,  OH,  December  1977,  (AD  A056  502). 

Analysis  of  Cost  Performance  Report  (CPR)  information  based  on 
Bayesian  inference.  Made  separate  variance  predictions. 
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21.  Howard,  T.W.,  "Methodology  for  Developing  Total  Risk 
Assessing  Cost  Estimate  (TRACE)",  U.S.  Army  Logistics  Management 
Center,  Fort  Lee,  VA,  ALM-63-4476-H3,  November  1978,  (#). 

Instructional  pamphlet. 

22.  Husic,  F.J.,  Cost  Uncertainty  Analysis,  Research  Analysis 
Corporation,  McLean,  VA,  May  1967,  (AD  686  770). 

23.  Hutzler,  W.P.,  Nelson,  J.R.,  Rei,  R.Y.,  and  Francisco,  C.M., 
Non  Nuclear  Air  to  Surface  Ordnance  For  The  Future:  An  Approach 
to  Propulsion  Technology  Risk  Assessment,  The  RAND  Corporation, 
Santa  Monica,  CA,  October  1982,  (#). 

Time  of  arrival  regression  to  predict  magnitude  of  risk  in  con¬ 
ceptual  programs. 

*2 4. Hwang,  J.D.,  Analysis  of  Risk  for  the  Material  Acquisition 
Process,  Part  I  -  Fundamentals,  U.S.  Army  Armament  Command,  Rock 
Island  Arsenal,  Rock  Island,  IL,  1970;  (AD  715  394),  (LD  25933). 
Subjective  probability  method  is  choice  between  gambles  pdf/cdf 
and  Delphi;  presents  several  techniques  of  statistical  decision 
theory  and  subjective  judgement  collection;  also  decision  ana¬ 
lysis  methods  for  contract  definition  and  concepts  formulation 
phases . 

2 5 .  Hwang ,  J . D . ,  Analysis  of  Risk  for  the  Material  Acquisition 
Process,  Part  II  -  Utility  Theory,  U.S.  Army  Armament  Command, 
Rock  Island  Arsenal,  Rock  Island,  IL,  1971,  (ad  747365),  (LD 
25933A) . 

Presents  discussion  of  utility  theory,  lotteries  and  techniques 
to  elicit  utility  functions.  Subjective  probability  method  is 
choice  between  gambles  (pdf). 

26.  Hwang,  J.D.,  "Risk  Analysis  Versus  Systems  Analysis  in  the 
Materiel  Acquisition  Process",  Proceedings  of  the  Tenth  Annual 
United  States  Army  Operations  Research  Symposium,  26-28  May  1971, 
U.S.  Army  Research  Office,  Research  Triangle  Park,  NC,  1971,  (AD 
731  795)  ,  (LD  25933B) . 

27.  Hwang,  J.D.,  and  Banash,  R.C.,  An  Introduction  to  Decision/ 
Risk  Analysis,  U.S.  Army  Armament  Command,  Rock  Island,  IL,  1971, 
(LD  27240) . 

28.  Hwang,  J.D.,  and  Kodani,  H.M.,  An  Impact  Assessment 
Algorithm  for  R&D  Project  Risk  Analysis,  U.S.  Army  Air  Mobility 
R&D  Laboratory,  Ames  Research  Center,  Moffett  Field,  CA,  October 
1973,  (#). 
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Offers  approach  to  risk  analysis  data  collection.  Provides 
automated  algorithm  to  compute  success  probability  and  cost 
impact.  Unspecified  subjective  probability  method. 

29.  Hwang,  J.D.,  and  Shumway,  C.R.,  "Decision  Risk  Analysis  for 
Research  and  Development",  Proceedings  of  the  1972  U.S.  Army 
Operations  Research  Symposium —  "Risk  Analysis",  U.S.  Army 
Operations  Research  Of f ice-Durham,  Durham,  NC,  May  1972,  (AD  748 
407). 

30.  Jeas,  W.C.,  MAJ  USAP,  Development  of  Weapon  Systems:  A 

Process  of  Technical  Uncertainty  Identification  and  Resolution, 
Defense  Systems  Management  College,  Port  Belvoir,  VA,  1976,  (LD 
36396A). 

31.  Jordan,  H.R.,  and  Klein,  M.R.,  An  Application  of  Subjective 
Probabilities  to  the  Problem  of  Uncertainty  in  Cost  Analysis, 
Office  of  the  Chief  of  Naval  Operations,  Research  Analysis  Group, 
Systems  Analysis  Division,  Washington,  D.C.,  November  1975,  (AD 
A105  780) . 

Independence  of  cost  element  uncertainties  is  needed  in  this 
simulation  of  WBS  using  Beta  distributed  cost  element.  Docu¬ 
mentation  of  SPET  (Subjective  Probability  Estimation  Technique) 
Model . 

32.  Kerns,  W.R.,  "Risk  and  Uncertainty:  State  of  the  Art  in 

Application";  paper  presented  at  the  Federal  Acquisition  Research 
Symposium,  Washington,  D.C.,  1982,  (#). 

33.  Knight,  J.R.,  Comparison  of  PERT  and  PEKT/COST  with  RISCA 
and  RISNET  from  a  User-Manager  Standpoint,  U.S.  Army  Logistics 
Management  Center  (Florida  Institute  of  Technology),  Fort  Lee, 
VA,  1974,  (LD  31523). 

34.  Kraemer,  G.T.,  "A  Successful  Quantitative  Risk  Assessment 
Technique",  Martin,  Rowe,  Sherman,  Ed.  Proceedings;  Management 
of  Risk  and  Uncertainty  in  the  Acquisition  of  Major  Programs, 
University  of  Southern  California,  Colorado  Springs,  CO,  1981, 
(#) . 

Iterative  use  of  graphic  technique  for  functional  management 
control  and  contract  structuring. 

35.  Kraemer,  G.T.,  "Quick  and  Effective  Risk  Analysis", 
Transaction  of  the  AACE  Annual  Meeting,  21st,  Morgantown,  WV, 
1977,  (#). 

Quick,  effective  basic  method  using  normal  graph  paper. 
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36.  Larew,  R.E.,  "Decision  Making  in  Construction  Operations", 
in  Martin,  Rowe,  Sherman,  Ed.  Proceedings;  Management  of  Risk 
and  Uncertainty  in  the  Acquisition  of  Major  Programs;  University 
of  Southern  California,  Colorado  Springs,  CO,  1981,  (#). 

Attacks  common  distributions  recommends  using  the  RS  distribu¬ 
tion;  not  directly  usable  without  further  development. 

37.  Lieber,  R.S.,  "New  Approaches  for  Quantifying  Risk  and 

Determining  Sharing  Arrangements",  Paper  Presented  at  the  Federal 
Acquisition  Research  Symposium,  Washington,  D.C.,  1980,  (#). 

Basis  for  negotiating  share  and  for  risk  management.  PDF 
analysis . 

38.  Lochry,  R.R.  COL  USAF,  et  al,  Final  Report  of  the  USAF 
Academy  Risk  Analysis  Study  Team,  Deputy  for  Systems, 
Aeronautical  Systems  Division,  Wr ight-Patterson  AFB ,  OH,  August 
1971,  (AD  729  223). 

Subjective  probability  method — Delphi. 

39.  MacCrimmon,  K.R.,  and  Ryavec,  C.A.,  An  Analytical  Study  of 
the  PERT  Assumptions,  The  RAND  Corporation,  Santa  Monica,  CA, 
December  1962,  (AD  293  423). 

Critique  of  PERT  Assumptions. 

40.  McNichols,  G.R.,  "A  Procedure  for  Cost-Risk  Assessment",  in 

Martin,  Rowe,  Sherman,  Ed.  Proceedings:  Management  of  Risk  and 
Uncertainty  in  the  Acquisition  of  Major  Programs;  University  of 
Southern  California,  Colorado  Springs,  CO,  1981,  (#). 

Unspecified  subjective  probability  assessment  methods.  Good 
discussion  by  developer  of  method-of -moments  techniques. 

41.  McNichols,  G.R.,  "Independent  Parametric  Costing,  What? 

Why?  How?",  Proceedings  Spring  Conference  of  the  American 
Institute  of  Industrial  Engineering,  Pg .  3-11;  AIIE,  Norcross, 

GA,  1975; 

Examines  concepts  and  principles  of  independent  parametric 
costing  philosophy. 

42.  McNichols,  G.R.,  Hutzler,  W.P.,  Insley,  P.A.,  and  Atkinson, 
A.S.,  "Concurrency;  The  Program  Manager's  Dilemma",  Paper  Pre¬ 
sented  at  the  Federal  Acquisition  Research  Symposium,  Washington, 
D.C.,  1982,  (#). 

43.  Morris,  J.M.  and  D'Amore,  R.J.,  Aggregating  and  Communicat¬ 
ing  Uncertainty,  Rome  Air  Development  Center,  Griffiss  AFB,  NY, 
RADC-TR-80-113 ;  April  1980,  (AD  A086  987). 
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Subjective  probability  presentation.  Methods  of  producing 
well-founded  estimates.  Calibration  of  subjective  probability 
estimates.  Excellent  bibliography  on  subjective  probability. 

44.  Nelson,  J.R.,  Performance/Schedule/Cost  Tradeoffs  and  Risk 
Analysis  for  the  Acquisition  of  Aircraft  Turbine  Engines: 
Applications  of  R-1288-PR  Methodology;  The  RAND  Corporation, 
Santa  Monica,  CA,  June  1975,  (AD  A013  729). 

Technique  to  quantify  technical  advancement  and  to  trade  cost, 
performance  and  risk;  regression  basis. 

45.  Norton,  M. ,  Abeyta,  R.,  and  Grover,  P.,  Production _ Risk 

Assessing  Methodology  (PRAM),  U.S.  Armv  Materiel  Systems  Analysis 
Activity,  Fort  Lee,  VA,  1982,  (#). 

TRACE  to  minimize  disruptive  programming  in  pre-  and  early 
production  phase. 

46.  O'Flaherty,  J.,  Identification  and  Estimation  of  High  Cost 
or  High  Risk  Elements,  Research  Analysis  Corporation,  McLean,  VA, 
December  1980,  (AD  884  533). 

Rudimentary  discussion  of  WBS.  Discusses  level  of  detail  in 
WBS ' s . 

47.  Oraski,  H.C.,  et  al,  Acquisition  Cost  Estimating  Using 
Simulation,  Naval  Training  Equipment  Center,  Orlando,  FL, 
September  1975,  (AD  A  015  624). 

48.  Powell,  N.,  "Risk  Analysis  Methodology  for  Engineering 
Development  Contracts,"  Proceedings  of  the  Fourteenth  Annual 
United  States  Army  Operations  Research  Symposium  (APRS),  Vol.  I, 
U.S.  Army  of  Logistics  Center,  Fort  Lee,  VA,  (AD  B009  955L) . 

49.  Preliminary  Analysis  of  Technical  Risk  and  Cost  Uncertainty 

in  Selected  DARPA  Programs,  Meridian  Corporation,  Falls  Church, 
VA,  1981,  (AD  A107  402-Final  Report),  (AD  A103  792 — Interim 

Report ) ) . 

Complex  mix  of  decision  analysis  and  administrative  procedure. 

50.  Press,  S.J.,  and  Harman,  A.J.,  Methodology  for  Subjective 
Assessment  of  Technological  Advancement,  The  RAND  Corporation, 
Santa  Monica,  CA,  (R-1375),  1975,  (#). 

51.  Program  Appraisal  Methodology,  Air  Force  Armament  Systems 
Laboratory,  Eglin  AFB ,  FL,  ( AFATL-TF-79-72 ) ,  August  1979,  (AD 
B049  513L). 
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Risk  almost  peripheral  issue  in  multi-attribute  utility  model. 

52.  RISCA  3  Documentation,  U.S.  Army  Logistics  Management 
Center,  Fort  Lee,  VA,  ( ALM-63-3737-H2 ) ,  (#). 

53.  Risk  Model  Documentation,  Directorate  of  Cost  and  Management 
Analysis,  DCS/Comptroller ,  Air  Force  Systems  Command,  Andrews 
AFB ,  MD,  1  June  1981. 

54.  RISNET  Analyst's  Guide,  John  M.  Cockerham  and  Associates, 
Inc.,  Huntsville,  AL,  (#). 

55.  Robinson,  J.  N.,  CAPT  USAF,  Sequential  Probability  Ratio 
Tests  of  the  Scale  Parameter  Between  Two  Weibull  Distributions 
with  Known  Shape  Parameter,  Air  University,  Air  Force  Institute 
of  Technology,  Wr ight-Patterson  AFB,  OH,  December  1976,  (AD  A034 
999). 

A  research  experimental  design. 

56.  Rowe,  A.J.,  "Methods  to  Predict  Cost  Overruns  in  the 
Acquisition  Process",  Paper  Presented  at  the  Federal  Acquisition 
Research  Symposium,  Washington,  D.C.,  1982,  (#). 

Proposes  a  scale  for  quantifying  technical  complexity;  descrip¬ 
tive  overview. 

57.  Selman ,  J.H.N.,  and  Selman,  V.,  "Operations  Research  for 
Risk  Analysis  Evaluation",  Proceedings  of  the  1972  U.S.  Army 
Operations  Research  Symposium — Risk  Analysis,  U.S.  Army 
Operations  Research  Of f ice-Durham,  Durham,  NC,  May  1972,  (AD  748 
407)  . 

58.  Strauch,  R.,  Risk  Assessment  as  a  Subjective  Process,  The 
RAND  Corporation,  Santa  Monica,  CA,  1980,  (LD  48338A) . 

Discussion  of  need  to  couple  judgement  and  methodology  and 
to  use  models  suited  to  situation. 

*59. Sutherland,  W. ,  Fundamentals  of  Cost  Uncertainty  Analysis, 
Research  Analysis  Corporation,  McLean,  VA,  1971,  (AD  881  975). 
Fairly  good  but  elementary  treatment  of  mathematics  of  adding 
probabilities,  unspecified  subjective  probability  assessment 
method . 

60.  Thompson,  W.E.,  "Risk  Implications  for  Cost  Growth  in  Weapon 
System  Acquisition  Programs",  Concepts  -  The  Journal  of  Defense 
Systems  Acquisition  Management,  Volume  3,  Number  2;  Defense 
Systems  Management  College,  Fort  Belvoir ,  VA,  1982,  (#). 
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General  discussion  and  implications.  Unspecified  subjective 
probability  assessment  method. 

61.  Total  Risk  Assessing  Cost  Estimate  (TRACE)  Guide,  John  M. 
Cockerham  &  Associates,  Inc.,  Huntsville,  AL,  September  1979, 
(#) . 

62.  Varnell,  A.K.,  et  al,  "Risk  Analysis  in  Military  R&D 
Projects",  Proceedings  of  the  1972  U.S.  Army  Operations  Research 
Symposium — Risk  Analysis,  U.S.  Army  Operations  Research  Office- 
Durham,  Durham,  NC,  (AD  748  407). 

63.  Venture  Evaluation  and  Review  Technique  (VERT),  (ALM-63- 
5260-H) ,  U.S.  Army  Logistics  Management  Center,  Ft.  Lee,  VA, 
November  1979,  {#). 

See  also  G.L.  Moeller,  Venture  Evaluation  and  Review  Technique, 
Decision  Models  Directorate,  U.S.  Army  Armament  Material  Readi¬ 
ness  Command,  Rock  Island,  IL,  November  1979,  (AD  076  600). 

64.  Venzke,  G.A.,  LTC  USA,  Implementation  of  Risk  Assessment  in 
the  Total  Risk  Assessing  Cost  Estimate  (TRACE),  U.S.  Army  War 
College,  Carlisle  Barracks,  PA,  1977,  (LD  39206A) . 

*65. Wilder,  J.J.,  An  Analytical  Method  for  Cost  Risk  Analysis, 
Grumman  Aerospace  Corporation,  Bethpage,  NY,  (PDR-OT-T77-12) , 
1977,  (#). 

Quick  but  effective.  Quantifies  uncertainty  by  generating 
moments  and  estimating  bounded  error. 

66.  Wilder,  J.J.,  and  Black,  R, ,  "Using  Moments  in  Cost  Risk 
Analysis",  in  Martin,  Rowe,  Sherman,  Ed,  Proceedings:  Manage¬ 
ment  of  Risk  and  Uncertainty  in  the  Acquisition  of  Major 
Programs ,  University  of  Southern  California,  Colorado  Springs, 
CO,  1981,  (#). 

67.  Wilder,  J.J.,  and  Black,  R.L.,  "Determining  Cost  Uncertainty 
in  Bottoms-Up  Estimating",  Paper  Presented  at  Federal  Acquisition 
Research  Symposium,  Washington,  DC,  1982,  (#). 

Dependency  treated.  Triangular,  uniform  and  beta  distribution 
treated . 

68.  Williams,  J.B.,  CAPT  USAF,  An  Analysis  of  Risk  Assessment 
Within  Aeronautical  Systems  Division,  The  School  of  Systems  and 
Logistics,  Air  Force  Institute  of  Technology,  Wright-Patterson 
AFB ,  OH,  1971,  (LD  27458). 
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69.  Worm,  G.H.,  "An  Application  of  Risk  Analysis  -  Uses,  Prob¬ 
lems  and  Pitfalls" , in  Martin,  Rowe,  Sherman,  Ed.  Proceedings: 
Management  of  Risk  and  Uncertainty  in  the  Acquisition  of  Major 
Programs ,  University  of  Southern  California,  Colorado  Springs, 
CO,  1981,  (#). 

70.  Worm,  G.H.,  Application  of  Risk  Analysis  in  the  Acquisition 
of  Major  Weapon  Systems,  Clemson  University,  Department  of 
Mathematical  Sciences,  Clemson,  SC,  1980,  (LD  49124A) . 

Easily  used,  computer  supported  method  for  determining  contract 
incentives , 

71.  Zettler,  W.T.,  MAJ  USAF,  Capital  Budgeting  Decisions:  A 
Study  of  Theory  Versus  Application,  Air  Command  and  Staff 
College,  Air  University,  Maxwell  AFB,  AL,  April  1979,  (AD 
B039  480L) . 

Research  on  utilization  of  capital  budgeting.  Methods  include 
types  of  risk  analysis. 

72.  Zschau,  E.V.W.,  Project  Modelling:  A  Technique  for  Esti¬ 
mating  Time-Cost-Performance  Trade-Offs  In  System  Development 
Projects,  The  RAND  Corporation,  Santa  Monica,  CA,  July  1969,  (AD 
691  810). 

Based  on  mathematical  programming,  and  utilizes  dual  variable 
approach . 
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THEORETICAL  DISCUSSION 


1.  Asher,  N.J.,  and  Maggelet,  T.F.,  On  Estimating  the  Cost 
Growth  of  Weapon  Systems ,  Institute  for  Defense  Analysis, 
Arlington,  VA,  June  1980,  (AD  A094  693),  (LD  49447A) . 

2.  Atzinger,  E.,  et  al.  Compendium  on  Risk  Analysis  Techniques, 
DARCOM  Materiel  Systems  Analysis  Activitiy ,  Aberdeen  Proving 
Ground,  MD,  1972,  (AD  746  245),  (LD  28463).. 

Subjective  probability  methods  are:  Choice  between  gambles 
pdf/cdf,  standard  lottery,  Churchman/  Ackoff,  Delphi. 

3.  Babiarz,  A. A.,  CAPT  USAF,  and  Giedras,  P.W.,  CAPT  USAF,  A 
Model  To  Predict  Final  Cost  Growth  in  a  Weapon  System 
Development  Program,  The  School  of  Systems  and  Logistics,  Air 
Force  Institute  of  Technology,  Wright-Patt.erson  AFB,  OH,  1975, 
(AD  A  016  040),  (LD  34803A) . 

Replicates  Glover-Lenz  application  of  Martin  Cost  Model  which 
incorporates  uncertainty  and  cost  analysis  addressing  final 
weapon  development  cost.  Measures  uncertainty  by  flatness  of 
probability  distribution,  defined  as  entropy. 

4.  Barclay,  Brown,  Kelly,  Peterson,  Phillips,  and  Selvidge, 
Handbook  for  Decision  Analysis,  Decisions  and  Designs,  Inc., 
McLean,  VA,  1977,  (#) . 

Provides  introduction  to  basic  concepts  and  operations  of  deci¬ 
sion  analysis;  includes  Bayesian  methods. 

5.  Bevelhymer,  H.L.,  CAPT  USAF,  A  Proposed  Metho'ology  for 
Weapon  System  Development  Risk  Assessment,  School  of  Engineering, 
Air  Force  Institute  of  Technology,  Wright-Patterscn  AFB,  OH, 
1973,  (AD  766  885),  (LD  29823). 

Developed  and  tested  methodology  for  obtaining  input  distribu¬ 
tions  from  routine  contractor-supplied  data.  Used  VERT  model 
to  develop  network.  Applied  method-of -moments .  Tested  metho¬ 
dology  on  actual  weapon  system  in  FSD .  Results  inconclusive, 
but  merit  further  application. 

6.  Chervaney,  N.L.,  et  al,  Analysis  and  Design  of  Computer- 
Based  Management  Information  Systems:  An  Evaluation  of  Risk 
Analysis  Decision  Aids,  University  of  Minnesota,  Minneapolis,  MN, 
September  1974,  (AD  A006  749). 

Research  on  usefulness  of  risk  analysis  decision  aids. 

7.  Dienemann,  P.,  Estimating  Cost  Uncertainty  Using  Monte  Carlo 
Techniques,  The  RAND  Corporation,  Santa  Monica,  CA,  January  1966, 
(AD  629  082) . 

8.  Donnell,  M.L.,  and  Ulvila,  J.W.,  Decision  Analysis  of  Advanced 
Scout  Helicopter  Candidates;  Decisons  and  Designs,  Inc.,  McLean, 
VA,  February  1980,  (AD  081  483). 

Decision  analysis  applied  to  technical  alternative  selection 
using  multi-attribute  utility  analysis,  but  no  explicit  con¬ 
sideration  of  risk. 
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9.  Edwards,  W.,  John,  R.,  and  Stillwell,  W. ,  Research  on  the 
Technology  of  Inference  and  Decision,  Social  Science  Research 
Institute,  University  of  Southern  California,  Los  Angeles,  CA, 
November  1977,  (AD  A056  921). 

Graphic  subjective  probability  assessment  method. 

10.  Graves,  S.B.,  A  Monte  Carlo  Risk  Analysis  of  Life  Cycle 
Cost,  Air  Force  Institute  of  Technology,  Wright-Patterson  AFB, 
OH,  September  1975,  (AD  A021  677). 

Reliability  and  maintainability  measures  used  as  variables 
showing  technical  risk  share  ratio,  award  fee  determinations. 

11.  Hayes,  R.A.,  CAPT  USAF,  An  Evaluation  of  a  Bayesian  Approach 
to  Compute  Estimates-at-Completion  for  Weapon  Systems  Programs, 
Air  University,  Air  Force  Institute  of  Technology,  Wright- 
Patterson  AFB,  OH,  December  1977,  (AD  A056  502). 

Analysis  of  Cost  Performance  Report  (CPR)  and  information 
based  on  Bayesian  inference.  Made  separate  variance  predic¬ 
tions. 

12.  Hwang,  J.D.,  Analysis  of  Risk  for  the  Material  Acquisition 
Process,  Part  I  -  Fundamentals,  U.S.  Army  Armament  Command,  Rock 
Island  Arsenal,  Rock  Island,  IL,  1970,  (AD  715  394),  (LD  25933). 

Subjective  probability  method  is  choice  between  gambles  pdf/cdf 
and  Delphi.  Presents  several  techniques  of  statistical  deci¬ 
sion  tneorv  and  subjective  judgement  collection.  Risk  analyses 
methods  of  contract  definition  and  concepts  formulation  phases. 

13.  Hwang,  J.D.,  Analysis  of  Risk  for  the  Material  Acquisition 
Process,  Part  II  -  Utility  Theory,  U.S.  Army  Armament  Command, 
Rock  Island  Arsenal,  Rock  Island,  IL,  1971,  (AD  747365),  (LD 
25933A) . 

Presents  discussion  of  utility  theory,  lotteries  and  tech¬ 
niques  to  elicit  utility  functions.  Subjective  probability 
method  is  choice  between  gambles  pdf. 

14.  Hwang,  J.D.,  and  Kodani,  H.M.,  An  Impact  Assessment 
Algorithm  for  R&D  Project  Risk  Analysis,  U.S.  Army  Air  Mobility 
R&D  Laboratory,  Ames  Research  Center,  Moffett  Field,  CA,  October 
1973,  (#). 

Offers  approach  to  risk  analysis  data  collection,  provides 
automated  algorithm  to  compute  success  probability  and  cost 
impact.  Unspecified  subjective  probability  assessment  method. 

15.  Larew,  R.E.;  "Decision  Making  in  Construction  Operations", 
in  Martin,  Rowe,  Sherman,  Ed.  Proceedings:  Management  of  Risk 
and  Uncertainty  in  the  Acquisition  of  Major  Programs;  University 
of  Southern  California,  Colorado  Springs,  CO,  1981,  (#). 

Attacks  common  distributions.  Recommends  using  the  RS 
distribution. 
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16.  Martin,  M.D.,  A  Conceptual  Cost  Model  for  Uncertainty  Para¬ 
meters  Affecting  Negotiated,  Sole-Source,  Development  Contracts, 
University  of  Oklahoma,  Norman,  OK,  1971,  (AD  A035  482^,  (LD 
37971A) . 

17.  McNichols,  G.R.,  "Macro  Models  For  the  Treatment  of  Uncer¬ 
tainty  in  Parametric  Costing",  Proceedings  Ninth  Annual  Meeting, 
Southeastern  Chapter,  Institute  of  Management  Sciences,  Pg.  57- 
66,  Clemson  University,  Clemson,  SC,  1973,  (#). 

Discusses  use  of  method  of  moments  in  macro  models;  subjective 
probability  assessment  method  used  is  unspecified  interview. 

18.  McNichols,  G.R.,  "Independent  Parametric  Costing,  What? 

Why?  How?",  Proceedings  Spring  Conference  of  the  American 
Institute  of  Industrial  Engineering,  Pg .  3-11;  AIIE,  Norcross, 

GA,  1975,  (#). 

Examines  concepts  and  principals  of  independent  parametric 
costing  philosophy. 

19.  McNichols,  G.R.,  On  the  Treatment  of  Uncertainty  in  Para¬ 
metric  Costing,  The  School  of  Engineering  and  Applied  Science, 

The  George  Washington  University,  Washington,  D.C.,  February 
1976,  (#). 

The  seminal  work  on  method  of  moments  in  parametric  costing. 

20.  McNichols,  G.R.,  "Treatment  of  Uncertainty  in  Life  Cycle 

Costing",  Proceedings  1979  Annual  Reliability  and  Maintainability 
Symposium,  IEEE,  1979,  (#). 

21.  McNichols,  G.R.,  "Uncertainties  of  LCC  Predictions",  Paper 

Presented  at  NATO  Advanced  Study  Institute  on  Electronic  System 
Effectiveness  and  Life  Cycle  Costing,  Norwich,  England,  1982, 
(#)  . 

Analytical  method  for  aggregating  LCC  component  and  subsystem 
costs . 

22.  Montgomery,  D.C.,  Callahan,  L.G.,  and  Wadsworth,  H.M., 
Application  of  Decision/Risk  Analysis  in  Operational  Tests  and 
Evaluation,  The  School  of  Industrial  and  Systems  Engineering, 
Georgia  Institute  of  Technology,  Atlanta,  GA,  September  1975, 
(AD  A024  205) . 

Report  on  four  theses.  Last  shows  novel  linear  programming 
approach.  Unspecified  subjective  probability  assessment  meth¬ 
ods  . 

23.  Moskowitz,  H.,  and  Sarin,  R.K.,  "Improving  Conditional  Prob¬ 
ability  Assessment  for  Forecasting  and  Decision  Making  in  Weapon 
System  Acquisition",  in  Martin,  Rowe,  Sherman,  Ed.  Proceedings: 
Management  of  Risk  and  Uncertainty  in  the  Acquisition  of  Major 
Programs ,  University  of  Southern  California,  Colorado  Springs, 
CO,  1981,  (#). 
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24.  Robinson,  J.  N.,  CAPT  USAP,  Sequential  Probability  Ratio 

Tests*  of  the  Scale  Parameter  Between  Two  Weibull  Distributions 
with  ^Known  Shape  Parameter,  Air  University,  Air  Force  Institute 
of  Technology,  Wright-Patterson  AFB,  OH,  December  1976,  (AD 
A034  999). 

A  research  experimental  design. 

25.  Rowe,  A.J.,  "Methods  to  Predict  Cost  Overruns  in  the 

Acquisition  Process",  Paper  presented  at  the  Federal  Acquisition 
Research  Symposium,  Washington,  D.C.,  1982,  (#). 

Proposes  a  scale  for  quantifying  technical  complexity. 
Descriptive  overview. 

26.  Sutherland,  W.,  Adding  Cost  Estimates  That  are  Not  Symmetric 
About  the  Most  Likely  Value,  Research  Analysis  Corporation, 
McLean,  VA,  1971,  (AD  883  232). 

Method  for  summing  estimates  without  a  computer  that  include 
high  and  low  estimates. 

27.  Sutherland,  W. ,  A  Method  for  Combining  Asymmetric  Three- 

Valued  Predictions  of  Time  or  Cost,  Research  Analysis  Corpora¬ 

tion,  McLean,  VA,  July  1972,  (AD  745  404). 

Graphically  derives  moments  of  Weibull  distribution  based  on 
asymmetry  coefficient. 

28.  Timson,  F.S.  and  Tihansky,  D.P.,  Confidence  in  Estimated 
Airframe  Costs;  Uncertainty  Assessment  in  Aggregated  Predic¬ 
tions  ,  The  RAND  Corporation,  Santa  Monica,  CA,  October  1972,  (#). 

Unspecified  subjective  probability  assessment  method. 

29.  Wilder,  J.J.,  An  Analytical  Method  for  Cost  Risk  Analysis, 
Grumman  Aerospace  Corporation,  Bethpage,  NY,  ( PDR-OT-T77-12 ) , 
1977,  (#). 

Quick  but  effective;  quantifies  uncertainty  by  generating 
moments  and  estimating  error  bounds. 

30.  Wilder,  J.J.,  and  Black,  R.,  "Using  Moments  in  Cost  Risk 
Analysis",  in  Martin,  Rowe,  Sherman,  Ed.  Proceedings  Management 
of  Risk  and  Uncertainty  in  the  Acquisition  of  Major  Programs, 
University  of  Southern  California,  Colorado  Springs,  CO,  1981, 
(#)  . 

31.  Worm,  G.H.,  Application  of  Risk  Analysis  in  the  Acquisition 
of  Major  Weapon  Systems,  Clemson  University,  Department  of 
Mathematical  Sciences,  Clemson,  SC,  1980,  (LD  49124A) . 

Easily  used,  computer  supported  method  for  determining  contract 
incentives . 

3 2 .  Worm ,  G.H . ,  Applied  hisk  Analysis  With  Dependence  Among  Cost 
Components ,  Clemson  University,  Clemson,  SC,  1981,  (AD  A119  617). 
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CASE  STUDIES 


1 .  AN/TPQ-36  Mortar  Artillery  Locating  Radar  Transport  Configu¬ 
ration  Decision  Risk  Analysis,  Systems  Analysis  Office,  U.S.  Army 
Electronics  Command,  January  1975,  (AD  B011  925L). 

Exhaustive  qualitative  application. 


2.  Air  Traffic  Management  Automated  Center  (ATMAC)  Concept  For¬ 
mulation  Study:  Computer  Model,  Volume  1,  Hughes  Aircraft  Com¬ 
pany,  Ground  Systems  Division,  Fullerton,  CA,  1974,  (AD  916 
524L) . 


3.  Amdor,  S.L.,  CAPT  USAF,  and  Kilgore,  R.R.,  CAPT  USAF,  Quanti¬ 
tative  Risk  Assessment:  A  Test  Case,  School  of  Engineering,  Air 
Force  Institute  of  Technology,  Wright-Patterson  AFB,  OH,  1974, 
(AD  777  585) ,  (LD  31450)  . 


4.  Babiarz,  A. A.,  CAPT  USAF,  and  Giedras,  P.W.,  CAPT  USAF,  A 
Model  To  Predict  Final  Cost  Growth  in  a  Weapon  System  Development 
Program,  The  School  of  Systems  and  Logistics,  Air  Force  Institute 
of  Technology,  Wright-Patterson  AFB,  OH,  1975,  i AD  A016  040),  (LD 
34803A). 

Replicates  Glover-Lenz  application  of  Martin  Cost  Model  which 
incorporates  uncertainty  and  cost  analysis  addressing  final 
weapon  development  cost.  Measures  uncertainty  by  flatness  ef 
probability  distribution,  defined  as  entropy. 


*5.  Barclay,  Brown,  Kelly,  Peterson,  Phillips,  and  Selvidge, 
Handbook  for  Decision  Analysis,  Decisions  and  Designs,  Inc., 
McLean,  VA,  1977,  (#). 

Provides  introduction  to  basic  concepts  and  operations  of  deci¬ 
sion  analysis.  Includes  Bayesian  methods. 


6.  Bevelhymer,  H.L.,  CAPT  USAF,  A  Proposed  Methodology  for 
Weapon  System  Development  Risk  Assessment,  School  of  Engineering, 
Air  Force  Institute  of  Technology,  Wright-Patterson  AFB,  OH, 
1973,  (AD  766885),  (LD  2 9823). 

Developed  and  tested  methodology  for  obtaining  input  distribu¬ 
tions  from  routine  contractor-supplied  data.  Used  VERT  model 
to  develop  network.  Applied  method  of  moments.  Tested  metho¬ 
dology  on  actual  weapon  system  in  FSD.  Results  inconclusive, 
but  merit  further  application. 


7.  Carodine,  F.,  Laube,  H.,  Esslinger,  W.H.,  Jr.,  and  Blue, 
D.L.,  Improved  Law  Cost  and  Schedule  Risk  Analysis,  Systems 
Analysis  Office,  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL, 
October  1975,  (AD  B010  641L). 

Standard  Decision  Risk  Analysis  (DRA)  for  Anti-Tank  Missile. 


8 .  Cost/Schedule  Risk  Analysis  of  Engineering  Development  Phase 
for  Army  User  Equipment  of  GPS,  ARINC  Research  Corporation,  Santa 
Ana,  CA,  April  1977,  (AD  A051  919). 
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Analysis  of  schedule  risk  taking  most  likely  times  from  Base¬ 
line  Cost  Estimate  but  for  maximum  and  minimum  using  assumed 
percentage  of  most  likely;  GPS  system — Example  of  derivative 
costing. 

9.  Cox,  L.,  and  Bohn,  M. ,  Report  on  the  Development  of  a  Proto¬ 
type  Computerized  Model  and  Data  Base  for  Use  in  Comparing 
Acquisition  Strategies,  The  Analytic  Sciences  Corporation, 
Arlington,  VA,  (TR-1375),  January  1981,  (#). 

Interactive  computer  model  which  compares  acquisition  strate¬ 
gies  based  on  the  weapon  system  concept  and  program  objectives. 

10.  Dodson,  E.N.,  "Risk  Analysis  in  the  Acquisition  of  BMD 
Systems , "  Proceedings  of  the  1972  U.S.  Army  Operation  Re¬ 
search  Symposium — Risk  Analysis,  U.S.  Army  Operations  Research 
Of f ice-Durham,  Durham,  NC,  May  1972,  (AD  748  407). 

11.  Donnell,  M.L.,  and  Ulvila,  J.W.,  Decision  Analysis  of  Ad¬ 
vanced  Scout  Helicopter  Candidates;  Decisons  and  Designs,  Inc., 
McLean,  VA,  February  1980,  (AD  081  483). 

Illustrates  use  of  decision  analysis  for  technical  alternative 
selection  using  multi-attribute  utility  theory,  but  no  explicit 
consideration  of  risk. 

12.  Glover,  W.L.,  CAPT  USAF,  and  Lenz,  J.O.  ,  CAPT  USAF,  A  Cost 
Growth  Model  for  Weapon  System  Development  Programs,  The  School 
of  Systems  and  Logistics,  Air  Force  Institute  of  Technology, 
Wright-Patterson  AFB,  OH,  1974,  (AD  785  438),  (LD  32006A) . 

Missile  cost  control  application.  Subjective  probability  meth¬ 
od  used  is  Delphi. 

13.  Graves,  S.B.,  A  Monte  Carlo  Analysis  of  Life  Cycle  Cost,  Air 
Force  Institute  of  Technology,  Wright-Patterson  AFB,  OH, 
September  1975  (AD  A021  677). 

Unspecified  subjective  probability  assessment  method. 

14.  Grayson,  A.S.,  CAPT  USAF,  and  Lanclos,  H.J.,  CAPT  USAF,  A 
Methodology  for  Subjective  Assessment  of  Probability  Distribu¬ 
tions,  The  School  of  Systems  and  Logistics,  Air  Force  Institute 
of  Technology,  Wright-Patterson  AFB,  OH,  1976,  (AD  A032  536),  (LD 
37757A) . 

Subjective  probability  assessment  and  content  analysis. 

15.  Hackenbruch,  D.J.,  and  VanHorn,  A.,  Decision  Risk  Analysis 
for  the  M-l  Tank  System,  Systems  and  Cost  Analysis  Directorate, 
U.S.  Army  Tank  Automative  Command,  July  1981,  (#). 

16.  Hackenbruch,  D.J.,  Initial  Operational  Capability  Schedule 
Risk  Analysis  for  Fighting  Vehicle  System,  Systems  and  Cost 
Analysis  Directorate,  U.S.  Army  Tank  Automative  Command,  March 
1981,  (#). 
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17.  Hackenbruch,  D.J.,  Risk  Assessment  of  Candidate  Mobile  Pro¬ 
tected  Gun  Systems,  Systems  and  Cost  Analysis  Directorate,  U.S. 
Army  Tank  Automative  Command,  May  1981,  (#). 

18.  Hayes,  R.A.,  CAPT  USAF,  An  Evaluation  of  a  Bayesian 
Approach  to  Compute  Estimates-at-Completion  for  Weapon  Systems 
Programs ,  Air  University,  Air  Force  Institute  of  Technology, 
Wright-Patterson  AFB,  OH,  December  1977,  (AD  A056  502). 

Analysis  of  Cost  Performance  Report  (CPR)  information  based  on 
Bayesian  inference.  Made  separate  variance  predictions. 

19.  Hutzler,  W.P.,  Nelson,  J.R.,  Rei,  R.Y.,  and  Francisco,  C.M., 
Non  Nuclear  Air  to  Surface  Ordnance  For  The  Future:  An  Approach 
to  Propulsion  Technology  Risk  Assessment,  The  RAND  Corporation, 
Santa  Monica,  CA,  October  1982. 

Time  of  arrival  regression  to  predict  magnitude  of  risk  in  con- 
ceputal  programs.  Risk  associated  with  achieving  technical 
advance  in  schedule  and  cost. 

20.  Hwang,  J.D.,  et  al,  A  Risk  Analysis  of  the  Improved  Cobra 
Armament  Program,  U.S.  Army  Air  Mobility  RfcD  Laboratory,  Ames 
Research  Center,  Moffett  Field,  CA,  June  1972,  (#). 

Unspecified  subjective  probability  assessment  method. 

21.  Mazza,  T.N.,  Paarman,  A.W. ,  and  Netzler,  M. ,  Risk  Analysis 
of  the  Army  Production  Plan  for  Self-Propelled  Howitzers,  U.S. 
Army  Armament  Command,  Systems  Analysis  Directorate,  Rock  Island, 
IL,  June  1976,  (AD  A026  681). 

Simple  network  defined  using  Baseline  Cost  Estimate  values. 
Analysis  method  not  stated,  subjective  probability  assessment 
used  is  centralized  decision  with  group  advice. 

22.  McGinnis,  J.P.,  LTC  USA,  and  Kirschbaum,  A. I.,  CAPT  USAF, 
"TRACE  Risk  Assessment  and  Program  Execution",  Defense  Systems 
Management  College,  Fort  Belvoir,  VA,  December  1981,  (#). 

Evaluation  of  TRACE.  Good  insight  on  experience  gained  in  use 
of  TRACE  since  inception. 

23.  Montgomery,  D.C.,  Callahan,  L.G.,  and  Wadsworth,  H.M., 
Application  of  Decision/Risk  Analysis  in  Operational  Tests  and 
Evaluation,  The  School  of  Industrial  and  Systems  Engineering, 
Georgia  Institute  of  Technology,  Atlanta,  GA,  September  1975, 
(AD  AO 2 4  205) . 

Report  on  four  theses.  Last  shows  novel  linear  programming 
approach.  Unspecified  subjective  probability  assessment 
methods . 

24 .  NASA/Army  XV-15  Tilt  Rotor  Research  Aircraft  Risk  Analysis, 
XV-15  Tilt  Rotor  Research  Aircraft  Project  Office,  NASA-Ames 
Research  Center,  Moffett  Field,  CA,  May  1974,  (#). 

Unspecified  subjective  probability  assessments  method. 
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25.  Netzler,  M. ,  Risk  Analysis  of  the  U.S.  Army  155  MM  Cannon- 
Launched  Guided  Projecticle  Program;  U.S.  Army  Materiel  Readiness 
Command,  Systems  Analysis  Directorate,  Rock  Island,  IL,  1974,  (AD 
A019  932)  ,  (AD  350  11A) . 

Simple  application  to  projectile  program. 

26.  Niemeyer,  W.A.,  Emanuel,  J.C.,  Kusterer,  E.T.,  and  Vegoda, 
R.J.,  Technical  Risk  of  Extended  Configurations  of  the  M113A1E1, 
U.S.  Army  Material  Systems  Analysis  Activity,  Aberdeen  Proving 
Ground,  MD,  July  1978,  (LD  42808A) . 

27.  Norton,  M. ,  Abeyta,  R.,  and  Grcver,  P.,  Production  Risk 
Assessing  Methodology  (PRAM),  U.S.  Army  Materiel  Systems  Analysis 
Activity,  Fort  Lee,  VA,  1982,  (#). 

TRACE  to  minimize  disruptive  reprogramming  in  pre-  and  early 
production  phases. 

28 .  Preliminary  Analysis  of  Technical  Risk  and  Cost  Uncertainty 
in  Selected  DARPA  Programs,  Meridian  Corporation,  Falls  Church, 
VA,  1981,  (AD  A107  402-Final  Report,  AD  A103  792-Interim  Report). 

Complex  mix  of  decision  analysis  and  administrative  procedure. 

29.  Relationship  Between  Cost  and  Schedule  Risk,  U.S.  Army 
Electronics  Command,  Systems  Analysis  Office,  Fort  Monmouth.  NJ, 
1975,  (LD  40117A) . 

Cost  risk  as  multiple  of  schedule  and  fixed  cost. 

30.  Risk  Analysis  of  a  V/STOL  Tilt  Rotor  Research  Aircraft,  U.S. 
Army  Air  Mobility  Research  and  Development  Laboratory  and  Tilt 
Rotor  Research  Aircraft  Projects  Office,  Ames  Research  Center, 
Moffett  Field,  CA,  March  1972,  (#). 

Unspecified  subjective  probability  assessment  method. 

31.  Robinson,  J.  N.,  CAPT  USAF,  Sequential  Probability  Ratio 
Tests  of  the  Scale  Parameter  Between  Two  Weibull  Distributions 
with  Known  Shape  Parameter,  Air  University,  Air  Force  Institute 
of  Technology,  Wright-Patterson  AFB,  OH,  December  1976,  (AD 
A034  999). 

A  research  experimental  design. 

32.  Seamands,  R.E.,  and  Hwang,  J.D.,  Analysis  of  Risk  for  the 
105MM,  Light,  Towed,  Soft  Recoil  Howitzer,  XM  204,  U.S.  Army 
Armament  Command,  Rock  Island  Arsenal,  Rock  Island,  IL,  1970. 

Unspecified  subjective  probability  assessment  method. 

33.  Shapiro,  P.  B.,  and  Kearley,  K.L.,  Technical  Report  78-13, 
Decision  Risk  Analysis;  COBRA  XM230E1  Program,  U.S.  Army  Troop 
and  Aviation  Readiness  Command,  Directorate  for  Plans  and  Systems 
Analysis,  Systems  Analysis  Division,  St.  Louis,  MO;  October  1978, 
(#)  . 

Investigates  risk  of  integrated  gun  into  airframe  using 
exhaustive  model  but  brief  analysis;  some  VERT  description. 
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34.  Smith,  J.A.,  Decision  Risk  Analysis  (DRA)  on  the 
Development  of  the  Bridge-Erection  Boat  for  the  Ribbon  Bridge, 
U.S.  Army  Mobility  Equipment  Research  and  Development  Center, 
Fort  Belvoir ,  VA,  May  1975,  (AD  B005  061L). 

Technical  alternative  selection.  Comparison  of  simple  acquisi¬ 
tion  strategy. 

35.  Tate,  R.  0.,  Remotely  Piloted  Vehicle  (RPV);  Management 
Reserve  for  the  Investment  Phase,  U.S.  Army  Aviation  Research  and 
Development  Command,  St.  Louis,  MO,  1981,  (AD  B061  737L) . 

36.  Thomas,  T.N.,  MAJ  USA,  VERT  -  A  Risk  Analysis  Technique  for 
Program  Managers,  Defense  Systems  Management  College,  Fort 
Belvoir,  VA,  1977,  (AD  A04  7620),  (LD  40483A) . 

37.  Timson,  F.S.,  Measurement  of  Technical  Performance  in  Weapon 
System  Development  Programs;  A  Subject  Probability  Approach, 
The  RAND  Corporation  (RM-5207-ARPA) ,  Santa  Monica,  CA,  1968,  (#). 

Subjective  probability  method  is  choice  between  gambles,  pdf/ 
cdf,  standard  lottery. 

38.  Tyburski,  D.,  Olson,.  H.,  and  Bernstein,  R. ,  Decision  Risk 
Analysis,  AN/TPQ-37  Artillery  Locating  Radar,  U.S.  Army  Electron¬ 
ics  Command,  Systems  Analysis  Office,  Fort  Monmouth,  NJ ,  1975, 
(LD  33186A) . 

Poor  documentation  but  fair  decision  aid. 


*  Core  Item. 

#  No  AD/LD  number  identified. 
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*1.  Antunes,  Moore,  Ed.,  Army  Programs  Decision  Risk  Analysis 
(DRA)  Handbook,  U.S.  Army  Logistics  Management  Center,  Fort  Lee, 
VA,  DARCOM  Handbook  11-1.1-79,  1979,  (#). 

Subjective  probability  methods  discussed:  Choice  between 

gambles  pdf/cdf,  standard  lottery,  Churchman/Ackof f ,  Delphi. 
Describes  role  of  DRA  in  material  acquisition  process,  as  well 
as  methods  of  performing  DRA's. 

2.  Edgar,  J.  D.,  LTC  USAF,  "Controlling  Murphy:  How  to  Budget 
for  Program  Risk",  Concepts  -  The  Journal  of  Defense  Systems 
Acquisition  Management,  Volume  5,  Number  3,  Defense  Systems 
Management  College,  Fort  Belvoir,  VA  1982,  (#). 

3.  Evriviades,  M. ,  Management  Reserve  Cost  Estimating  Relation¬ 
ship,  Cost  Estimating  and  Analysis  Division,  Directorate  of  Cost 
Analysis,  Comptroller,  Hanscom  AFB ,  MA,  March  1980,  {#). 

Determination  of  MR  Percent  at  ESD;  regression. 

4.  Howard,  T.W.,  "Methodology  for  Developing  Total  Risk 
Assessing  Cost  Estimate  (TRACE)",  U.S.  Army  Logistics  Management 
Center,  Ft.  Lee,  VA,  ALM-63-4476-H3 ,  November  1978,  (#). 

Instructional  pamphlet. 

*5.  Hwang,  J.D.,  Analysis  of  Risk  for  the  Material  Acquisition 
Process,  Part  I  -  Fundamentals,  U.S.  Army  Armament  Command,  Rock 
Island  Arsenal,  Rock  Island,  IL,  1970,  (AD  715  394),  (LD  25933). 
Subjective  probability  method  is  choice  between  gambles  pdf/cdf 
and  Delphi.  Presents  several  techniques  of  statistical  deci¬ 
sion  theory  and  subjective  judgement  collection.  Decision  ana¬ 
lysis  methods  of  contract  definition  and  concept  formulation 
phases . 

6.  McNichols,  G.R.,  "Cost-Risk  Procedures  for  Weapon  System 
Risk  Analysis",  1981  Proceedings:  Annual  Reliability  and  Mai.n- 
ainability  Symposium,  IEEE,  1981,  (#). 


7.  Program  Appraisal  Methodology,  Air  Force  Armament  Systems 
Laboratory,  Eglin  AFB,  FL,  AFATL-TF-79-72;  August  1979,  (AD  B049 
513) . 

Risk  almost  peripheral  issue  and  not  considered  in  TRACE-like 
context . 

8.  Total  Risk  Assessing  Cost  Estimate  (TRACE)  Guide?  John  M. 
Cockerham  &  Associates,  Inc.,  Huntsville,  AL,  September  1979, 
(#) . 


*  Core  Item. 

#  No  AD /LD  number  identified. 
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OTHER  TOPICS 


1.  Amdor,  S.L.,  CAPT  USAF,  and  Kilgore,  R.R.,  CAPT  USAF? 
Quantitative  Risk  Assessment:  A  Test  Case,  School  of  Engineer¬ 
ing,  Air  Force  Institute  of  Technology,  Wright-Patterson  AFB,  OH, 
March  1974,  (AD  777  585),  (LD  31450). 

2.  Fraker,  J.R.,  "Network  Models  for  Risk  Analysis",  in  Martin, 
Rowe,  Sherman,  Ed.  Proceedings;  Management  of  Risk  and  Un¬ 
certainty  in  the  Acquisition  of  Major  Programs,  University  of 
Southern  California,  Colorado  Springs,  CO,  1981,  (#). 

Short  background  on  two  network  software  sets  (RISCA  and  VERT) 
with  some  history. 

3.  Grover,  P.G.,  and  Schneickert,  G.D.,  MAJ  USA,  Total  Risk  As¬ 
sessing  Cost  Estimate  (TRACE):  A  Field  Survey,  Systems  and  Cost 
Analysis  Department,  School  of  Logistics  Science,  U.S.  Army 
Logistics  Management  Center,  Fort  Lee,  VA,  (#). 

Assessment  of  experience  in  using  and  teaching  TRACE  tech¬ 
niques  . 

4.  Lenox,  H.T.,  MAJ  USAF,  Risk  Assessment,  School  of  Engineering, 
Air  Force  Institute  of  Technology,  Wright-Patterson  AFB,  OH, 
1973,  (AD  767  871) . 

Includes  Bayesian  concepts. 

5.  Lilge,  R.W. ,  Total  Risk  Assessing  Cost  Estimate  (TRACE) ;  An 
Evaluation,  U.S.  Army  Research  and  Development  Command,  Systems 
and  Cost  Analysis  Division,  St.  Louis,  MO,  February  1979,  (AD 
B034  709)  ,  (LD  44065A) . 

Discussion  of  experience  in  AVRADCOM;  discusses  techniques  that 
could  be  used. 

6.  Lochry,  R.R.,  COL  USAF,  et  al,  Final  Report  of  the  USAF 
Academy  Risk  Analysis  Study  Team,  Deputy  for  Systems, 
Aeronautical  Systems  Division,  Wright-Patterson  AFB,  OH,  August 
1971,  (AD  729  223). 

Comprehensive  graphic  displays.  Methods  of  utilization  are 
covered. 

7.  McNichols,  G.R.,  "Generation  of  Parameter  Values  in  Sub¬ 
jective  Probability  Density  Functions",  Management  Consulting  and 
Research,  Inc.,  Falls  Church,  VA,  March  1977,  unpublished. 

8.  Williams,  C.,  and  Crawford,  F.,  Analysis  of  Subjective 
Judgement  Matrices,  The  RAND  Corporation,  Santa  Monica,  CA,  1980, 
(#) . 

Discusses  an  improved  method  for  obtaining  consistent  judgement 
matrices  from  subjective  estimates. 


#  No  AD/LD  number  identified. 
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APPENDIX  B 
DEFINITION  OF  TERMS 


Acquisition  Environment  -  The  totality  of  policies,  practices, 
and  practical  considerations  relative  to  management  of  acquisi¬ 
tion  programs. 

Activity  -  A  program  element  having  extension  in  time  and/or  con¬ 
suming  resources 


Arc  -  The  line  connecting  two  points  in  a  network. 

CER  -  Cost  Estimating  Relationship.  An  estimating  relationship 
in  which  cost  of  a  system  is  the  mathematical  result  of  a  formula 
having  selected  system  measurements  (like  thrust  or  weight)  as 
values  in  the  formula. 

Coefficient  of  Variation  -  Ratio  of  standard  deviation  to 
expected  value.  (See  Standard  Deviation  and  Expected  Value).  A 
measure  of  relative  uncertainty. 

Confidence  Interval  -  Limits  of  an  uncertain  quantity  (like  cost) 
between  which  there  is  a  given  probability  of  occurrence. 
Expressed  as  in  "the  n  percent  confidence  interval".  The  con¬ 
fidence  level  is  the  left  hand  lower  confidence  interval,  so  that 
one  may  say,  "C  is  the  nth  confidence  level",  meaning  there  is  an 
n  percent  probability  of  cost  being  between  0  and  C. 

Confidence  Level  -  Percentile 

Consistent  Judgement  Matrix  -  A  judgement  matrix  that  expresses 
relationships  like  probabilities,  so  that  if  probability  of  I  is 
m  times  that  of  J,  and  J  is  n  times  that  of  K,  then  the  proba¬ 
bility  of  I  is  mn  times  that  of  K.  Since  each  entry  is  a  ratio, 
rij,of  the  probability  of  I  divided  by  the  probability  of  J,  then 
r^j  times  rj^  equals  r^. 
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Critical  Patn  -  A  path  with  no  slack. 

Cumulative  Distribution  Function  -  A  curve  or  mathematical 
expression  which  associates  a  probability  to  all  values  in  the 
set  of  values  over  which  it  is  defined,  so  that  the  probability 
is  that  of  the  occurrence  of  a  value  less  than  or  equal  to  a 
given  value. 

Decision  Analysis  -  Examination  of  decision  problems  by  analysis 
of  the  outcomes  of  decision  alternatives,  the  probabilities  of 
arrival  at  those  outcomes,  and  the  intervening  decisions  between 
selection  of  alternatives  and  arrival  at  outcomes.  The  attribu¬ 
tes  of  the  outcomes  are  examined  and  numerically  matched  against 
preference  criteria. 

Decision  Tree  -  Representation  of  a  decision  problem  as  paths 
from  a  present  decision  through  alternative,  intermediate  deci¬ 
sions  and  risky  events  to  outcomes.  The  representation  is  simi¬ 
lar  to  an  increasingly  branched  tree. 

Engineering  Change  Order  Allowance  -  A  budget  category  to  be  used 
for  funding  changes  in  the  physical  or  performance  charac¬ 
teristics  of  a  system. 

Expected  Value  -  The  probabilistic  average  of  an  uncertain  quan¬ 
tity.  It  equals  the  sum  of  all  the  products  of  each  considered 
value  times  its  corresponding  probability.  Also  called  the  mean 
when  applied  to  all  possible  values  of  the  uncertain  quantity. 

Gantt  Chart  -  A  bar  graph  of  horizontal  bars  showing  program  ele¬ 
ment  commencement  and  termination  against  time. 

Histogram  -  A  vertical  bar  chart.  A  method  often  used  to  repre¬ 
sent  a  Probability  Mass  Function  (PMF). 

Incentive  Share  Ratio  -  The  ratio  of  government-to-contractor 
assumption  of  cost  or  saving  related  to  contract  target  cost. 


Independence  (also  statistical  independence)  -  The  relationship 
between  two  or  more  events  when  knowledge  of  the  probability  of 
occurrence  of  one  does  not  alter  the  probability  of  another. 

Judgement  Matrix  -  A  square  array  of  values  such  that  all  entries 
are  positive  and  for  every  entry  in  row  i  and  column  j  there  is 
an  entry  in  row  j  and  column  i  which  is  the  reciprocal  of  the 
first. 

Management  Reserve  -  An  amount  of  budget  held  aside  from  direct 
allocation  to  program  elements  as  a  reserve  for  contingencies. 

Mode  -  A  point  on  a  probability  density  function  where  the  proba¬ 
bility  goes  from  increasing  to  decreasing,  that  is,  a  maximum. 

Model  -  A  partial  description  of  a  system  using  sufficient  detail 
for  some  analytic  or  descriptive  purpose. 

Moment  -  A  function  (called  the  expectation)  of  a  probability 
law,  often  referred  to  as  an  "nth  moment",  where  n  is  any  number 
and  denotes  an  exponent  on  the  uncertain  quantity.  For  example, 
if  x  is  a  discrete  uncertain  quantity,  the  third  moment  is  the 
sum  of  all  values  of  x3  times  the  probability  of  each  respective 
value  of  x. 

Monte  Carlo  -  The  simulation  technique  in  which  outcomes  of 
events  are  determined  by  selecting  random  numbers  subject  to  a 
defined  probability  law.  If  the  random  number  falls  within  the 
limits  of  an  outcome's  probability,  that  outcome  is  chosen. 

Multiplicative  Cost  Element's  -  Cost  elements  whose  value  is 
derived  by  a  multiplication  of  other  cost  elements. 

Network  -  A  collection  of  points  connected  by  lines. 

Network  Program  Model  -  Representation  of  a  program  by  means  of  a 
network  in  which  the  points  (nodes)  stand  for  program  decision 
points  or  milestones  and  the  lines  (arcs)  stand  for  program  acti- 
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vities  which  extend  over  time  and  consume  resources.  Nodes  may 
be  regarded  as  activities  requiring  no  time  to  complete. 

Node  -  One  of  a  collection  of  points  defining  a  network. 

Objective  Probability  -  Probability  which  can  be  inferred  from 
objective  facts. 

Odds  -  The  ratio  of  probabilities  of  occurrence  and 
non-occurrence;  e.g.,  for  a  throw  of  a  fair  die  the  probability 
of  a  four  is  1/6.  The  odds  are  (1/6)  /  (5/6)  or  (1/5). 

Path  -  A  sequence  of  arcs. 

Percentile  -  The  value  of  an  uncertain  quantity,  generally 
referred  to  as  an  "nth  percentile",  which  is  greater  than  or 
equal  to  n  percent  of  all  values. 

PERT  -  Program  Evaluation  and  Review  Technique  .  An  early  net¬ 
work  analysis  technique  for  acquisition  programs,  in  which  each 
activity  duration  was  characterized  by  its  mean  or  expected 
values  and  no  uncertainties  were  incorporated. 

Probabilistic  Event  Analysis  -  Risk  assessment,  using  a  variation 
of  the  decision  analysis  method,  developed  in  reference  [54]  of 
Appendix  B,  Bibliography,  Basic  Discussion. 

Probability  Density  Function  (PDF)  -  A  probability  expression 
such  that  the  area  under  the  function  between  defined  limits  of 
the  values  on  which  it  is  defined  represents  the  probability  of 
the  values  within  those  limits. 

Probability  Function  -  A  mathematical  expression,  defined  on  an 
uncertain  quantity,  associating  a  probability  with  each  value  or 
non-redundant  combination  of  values  in  the  set. 

Probability  Mass  Function  (PMF)  -  A  function  assigning  probabili¬ 
ties  to  each  value  of  uncertain  quantity  having  only  discrete  or 
discontinuous  values. 


Program  Risk  -  The  probability  of  not  achieving  a  defined  cost, 
schedule,  or  technical  performance  goal. 

Random  Number  Generator  -  A  computer  program  capable  of  providing 
numbers  able  to  pass  statistical  tests  indicating  that  any  number 
between  the  limits  of  those  generated  is  equally  as  likely  to  be 
generated. 

Regression  Analysis  -  Determination  of  the  values  of  constants 
in  a  mathematical  expression  which  gives  results  that  are  the 
closest  to  the  observed  values  associated  with  values  of  the  data 
used  in  the  expression.  For  example,  if  cost  C  is  assumed  to  be 
the  sum  of  a  fixed  cost,  F,  and  variable  cost,  V,  for  N  items, 
C=F+VN.  If  data  show  the  expression  to  be  inexact,  regression 
analysis  finds  values  of  F  and  V  which  give  the  value,  C,  closest 
to  those  associated  with  all  data  values  of  N. 

Risk  -  The  condition  of  having  outcomes  with  known  probabilities 
of  occurrence,  not  certainty  of  occurrence. 

Risk  Assessment  -  Mathematical  analysis  of  the  probability  of 
achieving  or  not  achieving  acquisition  program  cost,  schedule,  or 
performance  goals. 

Simulation  -  The  operation  of  a  model  which  provides  outputs  ana¬ 
logous  to  the  system  modeled. 

Skew  -  The  asymmetry  of  a  probability  density  function.  The 
skew  is  to  the  side  of  the  mode  under  which  lies  the  greatest 
area. 


Skewness  -  The  measure  of  the  amount  of  skew. 

Slack  -  The  difference  between  the  earliest  possible  completion 
time  of  a  path  or  activity  and  its  latest  possible  completion 
time. 

Standard  Deviation  -  The  square  root  of  the  variance.  Often  used 
because  it  is  in  the  same  units  as  the  random  variable  itself, 


and  can  be  depicted  on  the  same  axes  as  the  PDF  of  which  it  is  a 
characteristic 


Standard  Normal  Function 
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_  A  probability  function  centered  on 

a  standard  deviation  of  1,  having  a  bell  shape  and 


zero,  with 

covering  values  that  become  negatively  and  positively  infinite. 


Subjective  Probability  -  An  expression  of  predictability  in  terms 
of  personal  statements  obeying  the  axioms  of  probability  and 
equal  to  the  probabilities  acceptable  to  the  assessor  for  a 
substitute  gamble. 

Uncertainty  -  The  condition  of  having  outcomes  with  unknown  prob¬ 
abilities  of  occurrence. 

Utility  Theory  -  Theory  of  preference  under  conditions  of  risk. 

Variance  -  A  measure  of  the  variability  of  a  random  variable. 
The  standard  deviation  squared.  Often  symbolized  as  Var  (  ) . 

Work  Breakdown  Structure  -  A  hierarchy  of  engineering  or  func¬ 
tional  elements  required  and  defined  by  reference  [15],  Chapter 
VI. 
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APPENDIX  C 


ACRONYMS 


ACAT 

AD/LD 


AFSARC 

AFSC 

ARB 

ASARC 

BCE 

CDF 

CDR 

CER 

DA 

DARCOM 

DCP 

DCSRDA 

DOD 

DPESO 

DRA 

DSARC 

DT 

EDT 

HQDA 

ICA 

IE 

ILS 

IPR 

ISR 

LOI 

MICOM 

MR 

MR 

NAVAIR 

NAVELEX 

NAVMAT 

NAVSEA 

NS  ARC 

OMB 

OPEVAL 

OSD 

OT 

PDF 

PEA 

PERT 

PM 


Acquisition  Category 

Document  number  prefix  for  documents  from  the 
Defense  Technical  Information  Center  and  the 
Defense  Logistics  Information  Exchange 
( respectively) 

Air  Force  Systems  Acquisition  Review  Council 
Air  Force  Systems  Command 
Acquisition  Review  Board 

Army  Systems  Acquisition  Review  Committee 
Baseline  Cost  Estimate 
Cumulative  Distribution  Function 
Commander 

Cost  Estimating  Relationship 
Department  of  the  Army 

U.S.  Army  Development  and  Readiness  Command 
Decision  Coordinating  Paper 

Deputy  Chief  of  Staff  for  Research,  Development, 
and  Acquisition 
Department  of  Defense 

Defense  Product  Engineering  Services  Office 
Decision  Risk  Analysis 

Defense  Systems  Acquisition  Review  Council 

Development  Test 

Engineering  Development  Test 

Headquarters,  Department  of  the  Army 

Independent  Cost  Analysis 

Independent  Estimate 

Integrated  Logistics  Support 

In  Process  Review 

Independent  Schedule  Review 

Letter  of  Instruction 

U.S.  Army  Missile  Command 

Management  Reserve 

Materiel  Readiness 

Naval  Air  Systems  Command 

Naval  Electronic  Systems  Command 

Naval  Material  Command 

Naval  Sea  Systems  Command 

Navy  Systems  Acquisition  Review  Council 

Office  of  Management  and  Budget 

Operational  Evaluation 

Office  of  the  Secretary  of  Defense 

Operational  Test 

Probability  Density  Function 

Probabilistic  Event  Analysis 

Program  Evaluation  and  Review  Technique 

Program  Manager 
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PMD 

- 

Program  Management  Directive 

PMF 

- 

Probability  Mass  Function 

PMO 

- 

Program  Management  Office 

POM 

- 

Program  Objectives  Memorandum 

PRR 

- 

Production  Readiness  Review 

R&D 

- 

Research  and  Development 

RA 

- 

Risk  Analysis 

RCE 

- 

Risk  Cost  Estimate 

RFM 

- 

Requiring  Financial  Manager 

SCP 

- 

System  Concept  Paper 

SSA 

- 

Source  Selection  Authority 

SSAC 

- 

Source  Selection  Advisory  Council 

STOG 

- 

Science  and  Technology  Objectives  Guide 

T&E 

- 

Test  and  Evaluation 

TECHEVAL 

- 

Technical  Evaluation 

TRACE 

- 

Total  Risk  Assessing  Cost  Estimate 

TRACE-P 

- 

Total  Risk  Assessing  Cost  Estimate  for 

USA 

- 

U . S .  Army 

USAF 

- 

U.S.  Air  Force 

USN 

- 

U.S.  Navy 

WBS 

- 

Work  Breakdown  Structure 
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APPENDIX  D 


QUANTIFYING  EXPERT  OPINION 

1.  GENERAL 

All  of  the  risk  assessment  methods  share  a  common  sensitive 
aspect  —  that  of  acquiring  quantitative  expressions  of  expert's 
uncertainty  that  violate  neither  their  own  beliefs  (are 
consistent)  nor  the  axioms  of  probability. 

By  "not  violating  the  axioms  of  probability"  is  meant  that 
the  probabilities  of  all  possible  events  must  sum  to  one,  that 
the  probability  of  any  event  must  be  a  number  between  zero  and 
one,  that  the  impossible  event  has  a  probability  of  zero,  and 
that  the  probabiltiy  of  joint  events  is  the  product  of  the  proba¬ 
bility  that  one  event  occurs  and  the  probability  that  the  other 
occurs,  given  that  the  first  has  occurred.  When  the  probability 
that  joint  events  occur  is  simply  the  product  of  the  probability 
of  each,  they  are  said  to  be  independent.  (See  Appendix  E  for 
more  on  independence) . 

Although  these  ideas  are  not  complex,  when  people  must  answer 
detailed  questions  about  probability,  an  analyst  often  finds  that 
even  mathematically  sophisticated  experts  violate  one  or  more  of 
the  principles.  In  order  to  forestall  misstatements  the  analyst 
should  be  familiar  with  a  number  of  techniques  of  obtaining  sub¬ 
jective  probability  estimates.  The  literature  of  decision  analy¬ 
sis  has  paid  particular  attention  to  these  problems  and 
techniques,  and  the  reader  can  find  interesting  discussions  in 
[2]  ,  [3]  ,  [4]  ,  and  [10]  . 

Figure  D-l  taken  from  [3]  helps  to  illustrate  the  need  for 
subjecting  uncertain  beliefs  to  the  discipline  of  quantification. 
Figure  D-l  shows  the  results  of  what  23  military  experts  meant  by 
the  phrases  shown  on  the  left  side  of  the  diagram.  All  the 
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Figure  D-l 

WHAT  UNCERTAINTY  STATEMENTS  MEAN  TO  DIFFERENT  READERS 
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experts  were  professionally  familiar  with  the  military  intelli¬ 
gence  context  in  which  these  expressions  were  used,  yet  their 
interpretations  in  quantitative  terms  ranged  over  the  values 
shown  where  the  dots  appear.  (The  dark  bars  represent  a 
researcher's  recommendations  for  standardizations).  Examples 
from  other  sources  show  similar  variation  in  intended  meanings. 

The  diagram  illustrates  the  need  for  a  common  language  in 
uncertain  situations;  but  how  can  the  analyst  undertake  to  pro¬ 
vide  this  common  language?  No  single  method  has  been  found  to  be 
completely  reliable,  but  a  number  of  different  methods  exist. 
The  use  of  two  or  more  methods  often  serves  to  detect  errors  and 
inconsistencies  which  can  then  be  examined  and  resolved. 

2.  DIRECT  METHODS 

A  first  approach  to  obtaining  subjective  probability  distri¬ 
butions  is  the  direct  one  of  requesting  the  estimate  of  a  proba¬ 
bility  to  be  assigned  to  each  value  of  a  range  of  values.  The 
method's  mirror  image  is  that  of  requesting  the  assignment  of  a 
value  to  a  selected  number  of  probability  levels,  such  as  zero, 
25  percent,  50  percent,  75  percent,  and  100  percent.  The  dif¬ 
ficulty  with  these  approaches  is  that  they  are  likely  to  produce 
results  that  are  intuitively  unappealing.  Because  of  the  way  the 
questions  are  phrased  and  because  of  the  confusion  likely  to 
exist  in  a  respondent's  mind,  the  result  is  likely  to  be  incon¬ 
sistent  or  to  violate  probability  axioms.  Figure  D-2  is  an  example 
in  which  total  probability  exceeds  one. 
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Figure  D-2 

INCONSISTENT  ASSESSED  PROBABILITY 


In  a  second  method,  the  analyst  requests  a  lowest  possible 
value,  a  most  likely  value,  and  a  highest  possible  value,  and 
either  makes  an  assumption  about  density  function  forms  or  asks 
his  expert  to  pick  a  form  from  an  illustrated  collection.  A  dif¬ 
ficulty  that  sometimes  occurs  with  this  method  is  that  experts 
have  at  times  been  unwilling  to  estimate  a  range  of  numbers  or 
anything  other  than  a  single  number.  Some  analysts  have  found 
some  respondents  willing  to  estimate  a  number  at  which  p  percent 
(where  p  might  equal  10,  or  5,  or  1)  of  the  values  will  be  less 
than,  more  than,  (for  the  high  end),  or  equal  to  the  given 
number,  as  in  Figure  D-3  where  1%  is  used. 


Figure  D-3 

PDF  CORRESPONDING  TO  A  THREE-VALUE  ESTIMATE 
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In  many  cases,  for  many  distributions,  the  latter  procedure 
can  imply  the  information  the  expert  was  unwilling  to  give 
directly.  Either  of  these  three-value  methods  may  be  adequate 
for  communicating  uncertainty  about  which  the  expert  respondent 
has  a  hazy  notion.  There  are,  however,  methods  available  that 
are  more  exact,  although  they  require  more  effort.  Some  of  these 
methods  are  discussed  below. 

3.  THE  MODIFIED  CHURCHMAN -ACKOFF  METHOD 

In  this  method,  discussed  in  [2],  the  expert  is  asked  to 
define  a  set  of  "events"  that  describe  the  outcomes  of  some 
"experiment".  For  example,  let  us  say  the  experiment  is  the 
development  of  a  WBS  element  and  the  events  are  the  costs  of 
development.  The  expert  must  define  a  set  of  events  that  is 
exclusive  (they  do  not  overlap)  and  exhaustive  (all  outcomes  are 
covered);  e.g.,  costs  from  10  to  15,  from  15.01  to  20,  from  20.01 
to  25,  and  from  25.01  to  30. 

This  set  of  events  is  exhaustive;  there  can  be  no  outcome 
below  15  or  greater  than  30.  It  is  exclusive  because  no  event 
overlaps  any  other — if  one  event  had  been  20  to  23  and  another  23 
to  25,  they  would  not  have  been  exclusive. 

Having  defined  a  set  of  events,  the  expert  roust  next  put  them 
in  order  of  likelihood.  For  example,  his  ordering  might  be  20  to 
25,  25.01  to  30,  15.01  to  20,  and  10.01  to  15.  If  any  are  of 
equal  likelihood,  that  must  be  stated. 

Next,  the  expert  is  asked  to  compare  each  event  against  each 
other  event  and  to  state  which  is  the  most  likely.  At  this  point 
the  analyst  should  graph  these  estimated  relative  magnitudes  as 
in  Figure  D-4.  The  redundant  exercises  are  carried  out  in  order 
to  detect  both  inconsistencies  and  equalities.  After  doing  this, 
the  events  are  again  ranked,  this  time  by  frequency  of  being  the 
more  likely  of  each  pair.  The  two  rankings  may  not  be  the  same, 
and  if  not ,  the  expert  should  be  asked  to  decide  which  ranking  he 
prefers . 
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Figure  D-4 

RELATIVE  RANKING  OF  EVENT  PROBABILITIES 

After  an  acceptable  ranking  has  been  obtained.,  the  expert 
should  be  asked  to  provide  ratios  of  the  event  pair  likelihoods. 
In  other  words,  he  is  asked  to  say  things  like,  “event  one  is 
two-thirds  as  likely  as  event  four",  where  event  one  may  be 
15.01-to-20,  and  four  may  be  20.01-to-25.  He  must  do  this  for 
every  pair  of  events.  By  assigning  an  arbitrary  value  to  any  one 
of  the  events,  the  sum  of  values  of  all  of  the  events  can  then  be 
set  to  equal  one.  By  using  the  ratio  of  the  value  of  each  event 
to  the  sum  of  the  values  of  all  events,  the  analyst  has  a  subjec¬ 
tive  probability  for  each. 


4.  THE  NORMALIZED  GEOMETRIC  MEAN  VECTOR  METHOD 


Experts  (and  laymen)  find  it  difficult  to  assign  consistent 
likelihood  ratios  (or  odds)  to  event-pairs.  Not  infrequently,  if 
there  are  a  large  number  of  event-pairs,  some  combination  of 
events  will  have  a  likelihood  with  which  the  providing  expert 
does  not  agree,  even  through  the  likelihood  was  derived  from  his 
ratio  statements.  The  authors  of  [14]  have  devised  a  method  to 
determine  the  assignment  of  individual  event  probabilities  which 
corresponds  most  closely  to  the  event  pair  statements,  yet 
fulfills  the  requirements  of  the  axioms  of  probability. 
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For  example,  if  there  are  five  events  under  consideration/ 
one  can  speak  of  the  likelihood  ratios  of  events  1-2,  1-3/  1-4/ 
and  1-5,  plus  the  ratios  of  events  2-3,  2-4,  and  2-5,  plus  those 
of  events  3-4  and  3-5,  plus  the  ratio  of  events  4-5.  Ratios  of 
the  likelihood  of  each  event  to  that  of  itself  are  1,  and  if 
there  is  a  ratio  n-m,  then  there  is  a  ration  m-n.  These  features 
make  a  matrix  a  natural  way  to  display  this  information,  as  in 
Figure  D-5. 
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Figure  D-5 
JUDGMENT  MATRIX 

Such  a  figure  meets  the  criteria  of  a  so-called 
"judgment-matrix"  in  that  the  entry  in  position  i,j  is  the 
reciprocal  of  the  entry  in  position  j,i  and  all  entries  are 
positive.  A  judgment  matrix  is  said  to  be  "consistent"  if  its 
entries  can  be  multiplied  so  that  entry  i,j  times  entry  j,k  gives 
entry  i,k. 

The  method  of  [14]  permits  taking  an  inconsistent  judgment 
matrix  and  deriving  from  it  a  set  of  probabilities  from  which  the 
closest  consistent  judgment  matrix  can  be  constructed.  The 
reader  who  wishes  to  know  what  "closest"  means  is  referred  to 
[14].  It  turns  out  that  the  probability  assignment  that  does  the 
job  is  the  normalized  geometric  mean  of  each  row;  that  is,  if  the 
matrix  compares  the  likelihoods  of  n  events  against  one  another 
and  there  is  therefore  an  n  by  n  sized  matrix,  the  probability  of 
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each  event  can  be  found  by  finding  the  product  of  the  ratios  in 
that  event's  row,  finding  the  nfch  root  of  that  product,  and 
dividing  by  the  sum  of  all  n  such  product  roots. 

To  illustrate,  take  the  following  matrix  as  representing  the 
assessed  likelihood  ratios  of  three  events. 
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Figure  D-6 

EXAMPLE  JUDGMENT  MATRIX 

Now  the  matrix  is  a  judgment  matrix,  but  it  is  not  con¬ 
sistent.  A  set  of  probabilties  that  will  yield  a  consistent 
matrix — one  that  will  represent  a  better  set  of  subjective 
probabilities — can  be  found  by  the  following  operations: 


Row 

1 

(1 

x  1/2  x  1/3) J/3 

0.5503 

Row 

2  . 

(2 

x  1  x  3/4 

= 

1.1447 

Row 

3 

(3 

x  3/4  x  1)  I'-* 

= 

1.5874 

Total 

3.2824 

P[EiJ 

= 

.5503/3.2824 

SS 

0.1677 

P[E2J 

= 

1.1447/3.2824 

= 

0.3487 

p[e3] 

1.5874/3.2824 

= 

0.4836 

These  probabilities  will  give  a  consistent  judgement  matrix 
as  close  to  the  original  as  possible,  as  shown  in  Figure  D-7; 

Event  1 _ 2 _ 3 _ 

1  1  .1677/. 3487  .1677/. 4836 

2  .3487/. 1677  1  .3487/. 4836 

3  .4836/. 1677  .4836/. 3487  1 

Figure  D-7 

CORRECTED  JUDGMENT  MATRIX 


or,  carrying  out  the  divisions,  Figure  D-8. 


Figure  D-8 

FINAL  JUDGMENT  MATRIX 

Comparing  Figure  D-8  with  Figure  D-6  shows  how  close  it  is,  in 
a  sense,  and  also  how  difficult  it  would  have  been  to  arrive  at  a 
consistent  set  of  probabilities  by  trial-and-error . 

The  Modified  Churchman-Ackof f  method  and  the  Normalized 
Geometric  Mean  method  will  give  the  same  answers  for  a  consistent 
judgment  matrix.  It  may  be  very  difficult  to  obtain  probabili¬ 
ties  resulting  in  a  consistent  matrix  by  direct  questioning,  and 
therefore,  by  use  of  the  Modified  Churchman-Ackof f  method.  A 
reasonable  approach  to  take  in  such  as  case  would  be  to  apply  the 
method  of  Normalized  Geometric  Means  to  obtain  a  probability 
distribution  and  then  to  see  if  the  expert  cannot  accept  it.  It 
is  possible  that  he  will  feel  his  ability  to  describe  his  own 
uncertainty  is  so  imprecise  that  he  can  accept  the  consistent 
distribution;  i.e.,  it  is  close  enough. 

5 .  GAMBLE  METHODS 

One  of  the  difficulties  inherent  in  either  of  the  two 
foregoing  methods  is  that,  as  the  number  of  relevant  events 
increases,  so  does  the  difficulty  of  directly  assigning  probabi¬ 
lities  or  ratios  of  probabilities.  For  example,  had  the  events 
in  these  previous  examples  been  defined  as  $1.00  intervals,  the 
number  of  events  would  have  been  too  great  to  evaluate  probabi¬ 
lity  ratios  meaningfully.  What  is  to  be  done,  then,  when  the 
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number  of  relevant  events  is  very  large?  Fortunately,  other 
methods  can  be  used,  and  some  are  discussed  in  the  following 
sections : 

a .  Choice  Between  Gambles  for  Probability  Density  Functions 
(CBG/PDF ) . 

Many  people,  when  questioned  one  way,  are  likely  to  make 
probability  statements  that  are  inconsistent  with  what  they  will 
say  when  questioned  in  another  way,  especially  if  they  are  asked 
for  direct  assignment  of  probabilities. 

One  method  of  phrasing  questions,  covered  in  [2]  and  [43, 
that  helps  avoid  inconsistency  states  the  problem  in  terms  of 
betting  gambles .  In  this  context  the  analyst  will  ask  the  expert 
to  consider  two  situations,  one  in  which  the  ucertain  event  will 
occur  or  not,  and  the  other  in  which  the  expert  will  participate 
in  betting  on  a  game  of  chance.  In  the  case  of  the  given  event, 
the  expert  is  told  he  will  receive  a  valuable  prize  if  the  event 
occurs.  In  the  case  of  the  game  of  chance,  he  will  receive  the 
same  prize  if  he  wins.  He  is  asked  to  name  the  odds  (or 
probability)  that  he  would  accept  in  the  game  of  chance  to  make 
him  indifferent  between  playing  the  game  or  receiving  his  prize 
in  the  case  of  the  event's  occurrence. 

To  complete  a  probability  density  function  (or  a  histogram), 
the  analyst  repeats  this  procedure  fcr  a  convenient  number  of 
events  over  the  range  of  the  uncertain  quantity  (e.g.,  cost).  In 
the  case  of  the  histogram  (Probability  Mass  Function)  he  is 
finished.  For  a  PDF  he  plots  the  points  determined  at  the  center 
of  the  range  of  each  event  and  smooths  in  a  curve,  so  that  the 
area  under  it  equals  1,  as  in  Figure  D-9.  The  analyst  must  also 
ensure  that  all  the  probability  theory  requirements  are  met  (see 
paragraph  1  for  those  requirements).  If  they  are  not  met, 
questions  must  be  reformulated,  or  numbers  revised  by  nor¬ 
malization,  or  some  combination. 
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Figure  D-9 

HISTOGRAM  AND  APPROXIMATE 
DENSITY  FUNCTION 

A  variant  of  this  method  uses  a  physical  analogy  rather  than 
simply  a  statement  of  odds  or  probability.  This  is  called  the 
Standard  Lottery  Method.  The  analogy  used  may  be  an  urn  with  a 
number  of  black  balls  and  a  number  of  white  balls,  the  sum  of  the 
two  colors  totaling  100.  The  probabiltiy  is  determined  when  the 
ratio  of  black  to  white  balls  is  such  that  the  expert  is  indif¬ 
ferent  between  receiving  the  prize  either  for  drawing  the  right 
colored  ball  or  for  occurrence  of  the  event. 


Another  analogy  is  the  dial  with  the  spinning  arrow.  The 
expert  is  asked  to  draw  the  sector  of  the  dial  which  would  make 
him  indifferent  to  receiving  the  prize  either  for  the  arrow's 
stopping  within  his  chosen  sector  or  for  the  occurrence  of  the 
event.  The  ratio  of  the  length  of  the  sector's  edge  to  the  full 
circle's  circumference  is  the  probability. 


D-ll 


The  last  analogy  is  for  the  expert  to  name  the  number  of  lot¬ 
tery  tickets  he  would  have  to  be  given  (each  ticket  equally 
likely  to  win)  out  of  a  total  number  for  him  to  be  indifferent 
between  receiving  the  prize  either  from  winning  the  lottery 
drawing  or  from  the  occurrence  of  the  event. 

b.  Choice  Between  Gambles  for  Cumulative  Distribution 
Functions  (CBG/CDF). 

In  this  method,  shown  in  [2]  and  [4],  the  analyst  holds  the 
probabilities  fixed  and  varies  the  size  of  the  event  being  eva¬ 
luated.  In  order  to  do  this,  the  analyst  asks  the  expert  to  name 
a  point  on  the  rang=  of  the  uncertain  quantities  at  which  he 
would  be  indifferent  between  receiving  a  prize  for  the  true  value 
to  fall  in  the  lower  (or  upper)  partial  range  and  receiving  a 
prize  for  correctly  calling  a  fair  coin-toss. 

Another  way  of  putting  the  question  is  to  ask  the  expert  to 
name  a  value  such  that  the  true  value  is  as  likely  to  be  less  as 
to  be  more  than  the  value  he  names.  How  this  information  is  con¬ 
verted  to  usable  values  will  be  explained  in  the  next  few 
paragraphs . 

The  next  step  in  this  procedure  is  to  ask  the  expert  to  name 
a  value  in  the  lower  (or  upper)  partial  range  at  which  he  is 
equally  as  willing  to  be  awarded  the  prize  for  the  true  value  to 
be  below  that  named  value  (and  not  in  the  range  between  the  named 
value  and  that  named  in  the  first  step)  or  for  correctly  calling 
a  fair  coin-toss. 

The  following  steps  repeat  this  process  as  many  times  as  are 
convenient  in  order  to  obtain  enough  points  to  draw  a  reasonably 
smooth  curve.  Usually  no  fewer  than  three  internal  points  are 
required,  plus  two  extreme  points.  The  extreme  point  at  the  low 
end  represent  the  highest  point  the  expert  can  name  which  he 
feels  certain  will  not  ever  exceed  the  true  value.  The  extreme 
point  at  the  high  end  represents  the  least  value  he  can  name 
which  he  is  certain  the  true  value  can  never  exceed. 
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The  product  of  the  CBG/CDF  is  a  cumulative  distribution  func¬ 
tion  similar  to  Figure  o-10. 


Cost 


Figure  D-10 


CUMULATIVE  DISTRIBUTION  FUNCTION 


Before  the  analyst  produces  the  CDF,  he  will  find  it  con¬ 
venient  to  use  an  interval  plot,  such  as  Figure  D-ll,  as  a  working 
tool.  The  following  approach  is  adapted  from  [4].  Figure  D-ll, 
Line  (a),  shows  how  the  interval  plot  would  look  upon  starting. 
The  first  step  would  establish  a  cost  value  which  we  will  call 
V50.  We  plot  it  as  in  Line  (b).  The  second  step  establishes  the 
point  where  the  expert  believes  there  is  as  much  chance  in  the 
true  value's  lying  to  its  left  as  there  is  in  its  lying  to  the 
right,  but  below  V50,  as  in  Line  (c)  .  This  point  is  called  V25* 
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The  point,  V75 ,  shown  in  Line  (d) ,  is  similarly  found.  Line  (e) 
finishes  the  working  plot  with  Vg  and  V^gg.  Each  of  the  points 
determined  represents  an  endpoint  to  an  interval  within  which  an 
equal  probability  is  "spread".  From  this  interval  plot  we  make  a 
CDF  by  starting  the  curve  on  the  horizontal  axis  at  Vg,  putting  a 
point  at  a  height  of  0.25  at  V25 ,  another  at  a  height  of  0.50  at 
Vsg,  and  so  forth.  The  method  of  breaking  up  intervals  produces 
a  CDF;  i.e.,  a  curve  that  means  the  true  value  of  the  uncertain 
cost  has  the  stated  probability  of  being  less  than  or  equal  to 
the  value  corresponding  to  the  probability. 
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Figure  D-ll 


INTERVAL  PLOTS 


From  this  CDF  the  analyst  can  draw  a  Probability  Density 
Function  (PDF)  to  check  it  for  inconsistency  with  the  expert's 
true  belief.  He  does  this  in  the  following  way. 
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Suppose  the  values  in  the  CDF  in  Figure  D-10  are  the  following 

V0  =  10 
V25  =  30 
V50  =  37 
V75  =  48 

v100  =  85 

Divide  the  curve  into  segments  in  which  the  slope  can  be 
closely  approximated  by  straight  lines.  Divide  the  vertical 
height  of  each  segment  by  the  horizontal  lengtn  of  the  segment. 
Plot  the  results  at  points  at  the  midpoints  of  the  horizontal 
segments.  Smooth  in  the  points.  Figure  D-ll  results  from  using 
this  procedure  with  Figure  D-12,  and  is  typical  of  such  approxi- 
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Figure  D-12 

PDF  CONSTRUCTED  FROM  SUBJECTIVE  CDF 


mate  PDF's.  It  does  not  look  as  smooth  as  a  classical  PDF  of 
probability  theory  and  it  probably  does  not  contain  an  area  of  1 
under  it,  but  it  gives  a  fair  idea  of  the  implications  of  the 
expert's  statements.  If  the  PDF  showed  a  "lumpier"  profile  the 
analyst  might  suspect  inconsistency  and  ask  the  expert  to  confirm 
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I  or  modify  his  statements.  The  analyst  also  might,  in  this  case, 

fe 

p  draw  a  triangular  or  other  classical  distributional  and  ask  the 

%  ,  t 
|  expert  whether  or  not  he  could  accept  it  as  an  approximation. 

|y 

|  Making  this  change  would  permit  much  more  convenient  mathematical 

6 

I  manipulation. 

t 

f  The  triangular  distribution  shown  in  Figure  D-13  results  from 

)  fitting  a  triangle  to  Vq,  V^qq  and  the  high  point  located  in  the 

|  same  place  as  that  in  Figure  D-12.  A  CDF  derived  from  the 

|  triangular  PDF  differs  from  the  original  (as  shown  in  Figure 

|  D-14)  by  amounts  that  might  be  considered  unacceptable,  so  the 

|  .  . 

I  fitting  of  a  triangle  may  be  inappropriate. 
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Figure  D-13 

TRIANGULAR  PDF  BASED  ON  SUBJECTIVE  CDF 


Another,  more  complex  distribution  like  that  in  Figure  D-15 
having  the  same  lower  and  upper  limits  and  high  point  location 
may  be  considered  more  appropriate.  This  curve  fitting  may  be 
accomplished  by  the  methods  of  [11]. 
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or  near  an  extreme  (the  PDF  is  symmetric  or  skewed,  respectively), 
that  there  are  or  are  not  limits  to  the  value  (the  PDF  hits  or 
does  not  hit  the  horizontal  axis),  or  that  he  is  quite  or  not 
very  uncertain  (the  PDF  has  a  large  or  small  relative  spread). 
Having  identified  a  PDF  shape,  the  expert  then  supplies  his  esti¬ 
mates  of  most  likely  and  extreme  values.  The  extreme  values  can 
either  be  absolute,  for  those  situations  in  which  the  PDF  hits 
the  horizontal  axis,  or  percentile  values,  for  kinds  where  it 
does  or  does  not  hit. 

The  attractiveness  of  the  diagrammatic  method  is  its  seeming 
ease  of  application.  Problems  can  arise,  however,  when  the 
expert  is  not  relatively  familiar  with  prooability  concepts.  For 
example,  maybe  his  true  PDF  simply  is  not  like  any  of  those 
shown,  then  one  of  the  other,  previous  methods  must  be  used. 

Summary .  None  of  the  methods  just  discussed  is  foolproof.  None 
should  be  the  sole  method  used.  The  analyst  should  search  for 
inconsistencies  and  try  to  reconcile  them.  Some  analysts  take  a 
rather  perfunctory  attitude  toward  the  risk  assessment  area 
involved  with  describing  the  elemental  uncertainties,  but  it 
should  be  realized  that  a  risk  assessment  describes  people's 
judgements,  and  people's  judgements  are  complex.  A  risk 
assessment  performed  without  the  exercise  of  care  in  obtaining 
its  basic  inputs  can  be  properly  criticized. 


7 .  GROUP  ASSESSMENTS 

In  matters  of  great  criticality,  the  manager  may  not  be 
satisfied  to  consult  a  single  expert  on  some  element  of  a  risk 
assessment.  He  may  feel  that  there  is  no  single  completely 
authoritative  person,  but  that  three  or  more  should  contribute 
assessments.  This  sounds  fine,  but  what  does  he  do  with  three 
experts  who  disagree?  He  can  do  such  things  as  averaging  or 
weighted  averaging,  but  there  seems  to  be  no  persuasive  argument 
for  so  doing. 


Committee  approaches  to  obtaining  a  group  assessment  have 
been  found  to  incorporate  problems  relating  to  interpersonal 
pressures  to  a  degree  that  caused  researchers  at  the  RAND 
Corporation  to  devise  a  method  avoiding  those  pressures r  the 
Delphi  technique  [71. 

The  Delphi  technique  has  become  well-known  in  management 
circles,  but  is  subject  to  misconception.  Too  often  the  term  is 
used  to  identify  a  committee  or  multiple-interview  process,  and 
these  do  not  share  the  advantages  of  the  technique. 

The  Delphi  technique  differs  from  other  methods  of  obtaining 
a  group  opinion  in  its  keeping  the  group's  members  apart  from  one 
another  in  order  to  reduce  irrelevant  interpersonal  influences. 
Properly  carried  out,  the  technique  is  facilitated  by  a 
moderator's  obtaining  each  panel  member's  opinion  (using,  for 
example,  the  methods  cited  above)  and  each  member's  reasons  for 
his  opinion.  The  moderator  then  reduces  the  opinions  and  reasons 
to  standard  statements  to  preserve  anonymity  and  shows  the  panel 
member  the  aggregated  opinions  of  the  other  panel  members  in  sta¬ 
tistical  terms.  He  provides  each  panel  member  with  the  reasons 
justifying  the  opinions  that  differ  with  his,  and  requests  a  re- 
evaluation  and  more  substantiation.  This  iterative  feeding  back 
continues  until  no  further  substantial  change  results.  At  this 
point,  the  moderator  takes  the  final  individual  opinions  and  com¬ 
putes  a  set  of  median  values  to  represent  the  group  opinion.  The 
median  value,  rather  than  the  average,  is  used  as  a  central  esti¬ 
mate  to  prevent  the  estimate  from  being  overly  influenced  by 
extreme  individual  values. 

The  Delphi  technique  is  time  consuming,  but  there  seems  to 
be  no  other  which  averts  the  dangers  inherent  in  other  group 
response  methods. 
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APPENDIX  E 


STATISTICAL  INDEPENDENCE 

A  general  difficulty  in  determining  uncertain  values  that  are 
functions  of  other  uncertain  values,  as  labor  costs  may  be  pro¬ 
ducts  of  wage  rates  and  hours,  is  the  need  for  probabilistic 
independence  of  the  input  uncertain  values.  Probability  theory 
makes  computations  relatively  straightforward  when  inputs  are 
probabilistically  independent;  that  is,  when  knowledge  of  the 
probability  of  any  uncertain  value  does  not  provide  information 
about  the  probability  of  another.  When  the  analyst  can  choose 
the  elements  of  his  mathematical  model  so  that  they  are  con¬ 
sidered  probabilistically  independent,  he  can  be  more  assured 
that  he  knows  how  to  handle  his  model. 

Much  work  has  been  done  with  models  constrained  to  consider 
only  probabilistically  independent  inputs,  but  the  actual  systems 
being  modeled  do  not  always  behave  in  such  a  simple  fashion.  It 
is  preferable,  therefore,  not  tc  be  limited  to  the  use  of  inde¬ 
pendent  inputs. 

To  illustrate  this,  consider  the  following  example.  It  is 
expected  that  engineering  labor  wage  rates  will  rise.  Is  it 
reasonable  to  say  that  this  knowledge  gives  no  information  on  the 
likely  behavior  of  other  wage  rates?  Certainly  some  correlation 
exists,  and  inputs  relating  to  those  matters  cannot  be  strictly 
independent. 

Usually  an  analyst  who  wishes  to  account  for  dependence  will 
try  to  use  an  aggregation  of  all  of  the  model  elements  which  are 
not  independent,  so  that  the  aggregate  itself  is  independent  of 
all  the  other  elements.  The  aggregated  values  collectively 
become  a  single  separate  and  independent  element. 

Other  ways  the  problem  of  dependence  can  be  overcome  to  a 
useful  degree  are  suggested  in  [3]  for  simulation  assessments  and 
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in  [10]  and  [13]  for  the  method  of  moments.  The  two  method~of~ 
moments  approaches  are  briefly  described  in  Appendix  F.  While 
more  complex  and  more  exact  methods  of  injecting  dependence  into 
simulations  (i.e.,  network  assessments  and  WBS  simulations)  are 
available,  they  will  not  be  discussed  herein. 
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APPENDIX  P 


TECHNIQUE  DESCRIPTIONS 

1.  GENERAL 

This  Appendix  examines  each  of  the  techniques  introduced  in 
Chapter  IV  in  greater  detail.  The  goal  is  for  the  reader  to 
acquire  an  appreciation  of  the  most  important  considerations  in 
accomplishing  assessments  by  use  of  each  individual  technique, 
however  the  reader  wishing  to  embark  on  an  assessment  will  need 
to  pursue  the  subject  in  the  references  for  the  specific  tech¬ 
niques  . 

2.  NETWORK  METHODS 

a .  Introduction 

The  reader  who  is  not  familiar  with  networks  and  the  terms 
PERT,  critical  path,  arc,  and  node  should  refer  to  any  of  the 
excellent  introductory  works  on  operations  research  such  as  [11] 
and  to  the  List  of  Terms,  Appendix  B. 

b .  The  Model 

In  general,  networks  like  Figure  F-l  consist  of  arcs  (lines) 
and  nodes  (end  points),  the  arcs  being  symbols  for  program  acti¬ 
vities  and  the  nodes  being  symbols  for  decision  points  at  the 
initiation  or  completion  of  the  activities.  Nodes  are  of  three 
types:  source  (or  originating)  nodes,  indicating  the  initiation 

of  the  program;  intermediate  nodes,  indicating  milestones  or  the 
initiation  and  termination  of  activities;  and  terminal  nodes, 
representing  the  completion  of  the  program  or  the  failure  to 
complete  some  segment  of  the  program.  Other  means  of  organizing 
networks  exist  but  will  not  be  discussed  here. 

An  awkwardly  large  comprehensive  network  can  be  partitioned 
into  "sub-networks"  when  computer  space  or  presentation  purposes 
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dictate.  Figure  F-l  shows  such  a  partition  and  the  resulting 
simplified  network.  For  example,  when  a  comprehensive  network  is 
too  large,  it  is  sometimes  possible  to  run  sub-networks  indepen¬ 
dently  and  to  use  the  derived  data  as  descriptors  of  arcs  in  a 
simplified  network  that  substitutes  for  the  original  sub-network. 


SUBNETWORK 


ACTIVITY  WITH 
BEHAVIOR  DETERMINED 


(b)  RESULTANT  EQUIVALENT  NETWORK 
Figure  F-l 
USE  OF  SUBNETWORKS 


While  these  characteristics  are  common  to  most  network  pro¬ 
ject  models,  the  techniques  with  which  we  are  particularly  con¬ 
cerned  must  use  "stochastic”  (probabilistic)  network  techniques. 
All  of  the  network  modeling  computer  programs  cited  in  this  hand¬ 
book  are  designed  to  model  probabilistic  networks.  In  a  probabi¬ 
listic  network  there  are  two  ways  in  which  uncertainty  manifests 
itself.  First,  activities  may  have  uncertain  outcomes  in  terms 
of  time  to  complete,  cost  to  complete,  or  achievement  of  one  or 
more  technical  performance  levels.  Second,  the  initiation  of 
activities  emanating  from  a  node  may  be  predictable  only  in  a 
probabilistic  way.  For  example,  a  test  outcome  (pass/fail)  may 


determine  whether  the  next  activity  is  a  progressive  continuation 
of  the  plan  or  a  corrective  action.  Since  the  test  outcome  can¬ 
not  be  predicted  with  certainty,  it  assumes  a  probabilistic 
nature.  The  network  model  represents  this  by  showing  at  least 
two  arcs  emanating  from  the  node  representing  test  activity 
completion.  The  analyst  assigns  each  arc  some  probability  of 
being  selected,  depending  on  the  information  he  has  collected 
from  experts.  Likewise,  he  assigns  probabilities  to  the 
appropriate  arcs  to  represent  the  relevant  probabilities  of 
completing  within  time  or  cost  constraints  or  of  meeting  perfor¬ 
mance  levels . 

An  important  aspect  of  network  models  that  is  needed  to  per¬ 
mit  realistic  simulation  of  programs  is  varied  "node  logic". 
"Node  logic"  refers  to  the  rules  which  determine  when,  for 
example,  a  decision  point  is  passed  and  when  it  initiates  the 
subsequent  activity. 

The  more  advanced  computer  programs  will  allow  use  of  both 
"AND"  and  "OR"  input  node  logic  and  ’DETERMINISTIC"  and 
"PROBABILISTIC"  output  node  logic.  The  two  types  of  input  logic 
determine  whether  all  (in  "AND"  logic)  or  only  one  (in  exclusive 
"OR"  logic)  or  some  (in  "OR"  logic)  of  tne  possible  arcs  entering 
a  node  must  be  completed  for  the  node  to  be  actuated.  The  two 
output  logics  determine  whether  all  (in  "DETERMINISTIC"  logic)  or 
only  one  (in  "PROBABILISTIC"  logic)  arc  (selected  at  random 
according  to  the  input  probabilities)  is  initiated  upon  comple¬ 
tion  of  node  actuation.  Although  later  versions  of  network  ana¬ 
lyzers  elaborate  on  these  rules  by  allowing  use  of  priorities  and 
other  elements,  these  are  the  basic  node  logic  concepts. 

A  principal  feature  of  the  stochastic  model's  handling  of 
both  nodes  and  arcs  is  its  capability  to  accept  a  variety  of 
PDF's.  One  program  (VERT)  [22]  can  accept  14  PDF's,  ranging  from 
the  well-known  uniform,  triangular,  and  normal,  through  the  more 
exotic,  such  as  log-normal,  Weibull,  and  Erlang.  If  none  of 


these  is  appropriate,  the  distribution  can  be  entered  as  a 
histogram  defined  by  the  analyst.  Any  or  all  can  be  used  in  a 
single  model.  Later  network  programs  also  accept  large  num¬ 
bers  of  PDF's. 

A  significant  decision  to  be  made  when  formulating  a  network 
model  is  the  determination  of  level  of  detail.  Most  analysts 
advise  the  completion  of  a  "rough-cut"  at  a  fairly  aggregated 
level  of  detail  before  attempting  to  model  the  fine  grained 
programmatic  structure.  This  allows  a  more  realistic  determina¬ 
tion  of  needed  level  of  detail  to  be  made  before  full  commitment, 
and  it  also  identifies  areas  which  will  need  to  be  emphasized  in 
the  future.  In  the  earliest  stages  of  program  definition  (e.g., 
concept  definition),  far  less  than  100  activities  can  be  included. 
Of  course,  the  inputs  are  estimates  of  more  highly  aggregated 
sets  of  activity  behaviors,  and  therefore,  they  will  contain  more 
inherent  uncertainty  than  will  be  the  case  as  program  definition 
progresses.  As  plans  are  defined  in  more  detail,  there  will  be  a 
need  to  define  and  include  more  activities;  however,  200-500  have 
been  found  adequate  by  most  analysts.  Their  consensus  for  analy¬ 
tical  (not  management  control)  purposes  is  that  greater  detail 
can  tend  to  obscure  major  relationships  and  slow  the  modeling 
process  unnecessarily. 

In  the  process  of  structuring  the  model,  analysts  do  not  go 
directly  from  activity  description  sheets  to  computer  terminals. 
Instead,  the  actual  networks  are  drawn.  Each  network  computer 
program  users'  manual  recommends  use  of  a  particular  set  of  sym¬ 
bols  that  define  the  logic  and  assist  the  analyst  in  ensuring 
inclusion  of  necessary  inputs.  Figure  F-2  illustrates  some  of 
these  symbols. 

The  work  of  formulating  a  network  model  should  not  be  ex¬ 
pected  to  proceed  without  some  difficulties.  Procedure  relation¬ 
ships  may  have  to  be  defined  many  times  as  inconsistencies  are 
discovered,  weaknesses  corrected,  and  budgets  or  schedules 
changed. 
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PREF 

Figure  F-2 
NODE  REPRESENTATION 

c .  Implementation 

When  the  network  is  finally  judged  to  be  "good  enough,"1  the 
analysts  enter  its  description  into  the  computer.  Specially  pre¬ 
pared  data  formats  peculiar  to  the  particular  network  analysis 
computer  program  are  used.  As  can  be  imagined,  the  data  set  may 
be  massive  since,  for  example,  in  some  network  models  a  single 
activity  can  require  as  many  as  9  descriptive  inputs,  and  a  node 
as  many  as  31. 

The  ’network  analyzers  cited  herein  provide  varying  amounts  of 
flexibility  and  realism  in  their  treatment  of  the  variables  of 


1A  model  is  an  abstraction  whose  realism  is  adequate  to  its 
purpose. 


time,  a  statistically  independent  quantity,  or  a  probabilistic 
function  of  time.  They  may  or  may  not  consider  technical  per¬ 
formance  as  an  uncertain  quantity.  Another  difference  is  in  the 
treatment  of  other  resources.  While  VERT,  for  example,  treats 
only  a  single  resource — cost — others  consider  a  user-defined 
number  (e.g.,  TRANSIM  V) [ 9 ] ,  [10]. 

When  the  analyst  is  ready  to  run  his  program  and  has  entered 
his  data  into  the  computer,  the  machine  performs  a  large  number 
of  simulations  of  the  execution  of  the  acquisition  program.  This 
is  assisted  by  use  of  a  computer  subprogram  known  as  a  "random 
number  generator",  which,  in  concept,  produces  a  number  that  lies 
anywhere  between  zero  and  one  with  uniform  probability.  Whenever 
an  uncertain  event  is  to  occur,  the  computer  program  compares 
that  number  with  a  data-generated  number  representing  the  proba¬ 
bility  of  the  event's  occurrence.  If  the  "random  number"  is 
within  the  range  of  the  data-generated  probability,  the  computer 
program  initiates  that  event.  Of  course,  no  single  simulation  of 
program  execution  conveys  much  useful  information;  however,  when 
that  simulation  is  complete,  the  computer  has  collected  data  from 
wherever  in  the  network  the  analysts  programmed  for  it  to  do  so. 
It  then  repeats  the  process,  typically  1,000  to  6,000  or  more 
times.  Obviously,  one  simulation  tells  very  little,  and  100  may 
still  not  give  "typical"  statistics.  How  many  repetitions  are 
enough?  There  is  no  practical  way  to  be  sure,  but  statistical 
methods  can  give  indications  of  what  is  reasonable.  Computer 
time  is  now  cheap  enough  so  that  analysts  can  increase  the  number 
of  repetitions  until  percentages  of  kinds  of  data  being  collected 
appear  to  have  stabilized.  The  1,000  to  6,000  replications  will 
consume  not  much  more  than  five  minutes  of  computer  time,  and 
statistical  tests  can  then  be  performed  to  see  if  the  results  are 
reasonably  stabilized. 

d .  Outputs 

Even  the  most  rudimentary  set  of  outputs  obtainable  from  net¬ 
work  analyzers  will  contain  terminal  node  information  like  PDF's, 
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Probability  function  graphics  produced  include  types  such  as 
Figures  F-4(a)  and  F-4(b)  below  for  selected  internal  nodes  (that 
is,  decision  points  in  the  program)  or  terminal  nodes  (ways  in 
which  the  program  can  end). 


PDF  CDF 

(a)  (b) 

Figure  F-4 

DISTRIBUTIONS  OF  COST  AT  A  TERMINAL  NODE 

Figure  F-3(a)  represents  an  approximate  PDF  (in  the  VERT 
manual  called  a  Relative  Frequency  Distribution)  and  Figure 
F-3(b),  a  CDF.  The  former's  shape  gives  intuitive  understanding 
of  probabilities  and  of  uncertainty,  whereas  the  latter’s  allows 
the  making  of  probability  statements  such  as,  "There  is  a  72  per¬ 
cent  probability  of  costs  being  at  or  under  $583,300".  The  sta¬ 
tistical  information  at  the  bottom  tells  the  number  of  times  the 
simulation  was  run  to  get  the  curves  (NO.  OBS),  which  was  999, 
the  expected,  or  mean,  cost,  which  was  $549,400,  and  certain  sta¬ 
tistics  describing  the  curves.  The  Standard  Error  gives  an  abso¬ 
lute  measure  of  variability  or  uncertainty  in  the  units  of  the 
variable  being  assessed  (here  it  is  cost),  while  the  Coefficient 
of  Variation  gives  a  relative  measure  since  it  is  the  ratio  of 
the  standard  error  to  the  mean.  A  value  near  1  for  the  coeffi¬ 
cient  of  variation  might  be  considered  to  show  fairly  high  uncer¬ 
tainty.  The  other  two  statistics  give  more  abstract  information. 
"Pearsonian  skew"  (i.e.,  skewness)  indicates  the  amount  the  PDF 
maximum  shifted  away  from  the  average,  and  "kurtosis"  tells  how 
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sharply  it  is  peaked.  Skewness  tells  a  manager  in  a  qualitative 
way  the  strength  of  belief  that  the  variable  will  be  different 
from  the  average.  He  will  obtain  meaningful  information,  also, 
by  looking  at  the  PDF  to  see  what  the  most  likely  value  is,  and 
then  Ly  looking  at  the  CDF  to  compare  the  probability  of  the 
value  being  below  or  equal  to  the  most  likely  and  below  or  equal 
to  the  average. 

By  looking  for  the  50  percent  point  (median)  on  the  CDF 
(shown  only  approximately  in  Figure  F-4(b))  he  can,  for  example, 
see  the  point  which  Army  TRACE  policy  says  should  be  used  for  a 
budget  and  compare  that  with  his  baseline  estimate's  probability. 

Other  outputs  of  VERT  consist  of  a  bar  graph  of  terminal  node 
probabilities.  Since  a  project  may  succeed  or  fail  in  a  number 
of  ways,  each  may  be  represented  by  a  terminal  node.  The  bar 
graph  gives  the  outcome  probabilities  and  allows  comparisons  to 
be  made.  The  program  can  even  select  an  optimum  terminal  node 
(that  which  has  least  cost  and  completion  time  and  highest  perfor¬ 
mance),  or  worst  terminal  node  (that  in  which  highest  cost  and 
completion  time  and  worst  performance  results.) 

In  many  analyzers,  this  information  is  supplemented  by  indi¬ 
ces  of  criticality  for  specified  arcs  and  nodes.  ("Index  of 
criticality"  means  the  probability  of  an  arc  or  a  node  being  on 
the  critical  path.) 2  Other  statistics  available  with  various 
analyzers  (e.g.,  RISNET  [23])  include  probability  of  reaching 
specific  terminal  nodes  (i.e.,  of  a  program's  ending  in  a  speci¬ 
fic  way)  and  joint  time  and  cost  distribution  (i.e.,  the  probabi¬ 
lity  that  final  specified  limits  on  both  time  and  cost  will  be 
exceeded)  . 

2it  should  be  remembered  that  in  probabilistic  networks,  dif¬ 
ferent  variables  and  paths  chosen  by  the  uncertain  nature  of 
the  program  will  cause  simulations  to  have  critical  paths 
which  are  not  necessarily  the  same  each  time.  See  Appendix 
B,  Definition  of  Terms  for  the  definition  of  "Critical  Path". 


Managers  can  use  these  outputs  for  decision  making  in  the 
following  ways.  Suppose  Figures  F-4(a)  and  F-4(b)  represent  the 
probability  density  function  and  cumulative  distribution  function 
of  cost  at  a  terminal  node,  which  itself  represents  completion  of 
an  important  milestone. 

Typically,  a  value  such  as  b  in  trie  two  figures  represents 
the  Baseline  Cost  Estimate  submitted  by  the  PMO;  but  Figure  F-4(a) 
clearly  shows  that  the  most  likely  cost  is  somewhat  higher,  at  m, 
and  that  the  expected  value  is  even  greater,  at  a. 3  Figure 
F-4(b)  shows  even  more  clearly  what  the  situation  is.  The  proba¬ 
bility  of  not  exceeding  b  is  only  pbf  while  the  cost  level  that 
has  an  even  chance  of  being  exceeded  or  not  is  Op  (statistically, 
the  median — also  the  level  of  the  TRACE).  Figure  F-4(b)  can  also 
be  used  to  set  a  cost  level  consistent  with  the  manager's  assess¬ 
ment  of  a  reasonable  level  of  risk.  For  example,  if  he  considers 
that  a  risk  of  failure  to  cover  costs  should  not  be  greater  than 
one  in  five  (the  20  percent  level),  he  can  set  his  budget 
requirement  at  the  80th  percentile  cost. 

A  flexible  computer  program  supporting  probabilistic  network 
assessment  is  TRANSIM  V  [9]  and  [10].  This  computer  model  lends 
itself  to  use  in  detailed  budgeting  and  day-to-day  project 
control.  The  utility  of  this  model  is  not  obtained  without 
effort  but  is  valuable  for  the  insight,  a  manager  can  have  into 
program  status  and  direction.  The  data  input  requirements, 
however,  expand  in  proportion  to  the  level  of  detail  in  the  out¬ 
put,  For  example,  for  each  activity  explicitly  described  in  the 
network  a  responsibility  is  named.  This  allows  summaries  by  name 
to  be  included  in  the  output  but  obviously  requires  more  data 
preparation , 

TRANSIM  V  provides  the  following  20  reports:  (Note  -  TRANSIM 
is  but  one  example  of  the  many  programs  available  that  analyze 
networks) . 

^"Expected  value"  is  defined  in  probability  as  the  average  for 
many  runs,  not  what  can  be  expected  to  occur.  The  latter 
would  be  a  prediction,  not  a  probability  statement. 


o  Time  Summary  Graph 


o  Criticality  Analysis  Report 
o  Activity  Criticality  Reoort 

-  With  Predecessors,  sorted  by  Activity /Milestone  Code 
and/or  Decreasing  Criticality 

-  With  Successors,  sorted  by  Activity /Milestone  Code 
and/or  Decreasing  Criticality 

o  Activities  Delayed  Awaiting  Resources 

o  Activity/Milestone  Probability  of  Occurrence  Report 

o  Activity /Milestone  Schedule,  sorted  by 

-  Activity /Milestone  Code 

and/or 

-  Earliest  Start  Times 

and/or 

-  Expected  Start  Times 

and/or 

-  Latest  Start  Times 

and/or 

-  Earliest  Finish  Times 

and/or 

-  Expected  Finish  Times 

and/or 

-  Latest  Finish  Times 
o  Activity  Schedule  Graph 

o  Activity  Code  and  Description  for  Schedule  Graph 
o  Resource  Utilisation  History 
o  Resource  Requirements  Graph 

-  Schedule 

-  Summary 

o  Activity  Code  and  Description  for  Resource  Requirements 
Graph 


F-ll 


o  Aggregate  Resource  History 
o  Aggregate  Resource  History  Graph 

-  Schedule 

-  Summary 

o  Activity  Code  and  Description  for  Aggregate  Resource 

History  Graph 

o  Cumulative  Aggregate  Resource  History  Graph 
o  Aggregate  Resources  Summary  Graph 
o  Activity  Status  Reports 

-  Activities  Underway  for  Responsibility 

-  Activities  Due  to  Start 

-  Activities  Due  to  Complete 

These  reports  can  be  printed  out  selectively.  To  describe  each 
report  in  detail  would  add  unnecessary  bulk  to  this  handbook.  It 
is  sufficient  to  summarize  as  follows: 

The  reports  provide  probability  of  completion  by  scheduled 
date  and  combine  the  information  with  actual  calendar  dates  when 
desired.  PDF's  and  CDF's  can  be  obtained  for  any  activity  or  deci¬ 
sion  point.  Designated  risk  levels  can  be  used  to  determine  when 
each  activity  is  expected  to  reach  that  level,  and  this  infor¬ 
mation  can  be  supplemented  by  a  reading  of  the  number  of  days 
between  reaching  that  level  and  the  date  of  scheduled  completion. 
As  in  other  network  models,  criticalities  (probabilities  of  being 
on  the  critical  path)  are  given,  but  TRANSIM  V  reports  also  pro¬ 
vide  summaries  by  responsibility.  An  evaluation  is  also  given  of 
program-wide  criticality,  which  the  program  manager  can  use  as  a 
general  status  evaluation.  Since  each  inputs  resource  is  limited 
(there  is  a  fixed  resource  pool  size)  during  any  activity,  and 
since  the  resource  cost  per  unit  for  each  activity  can  also  be 
defined,  the  manager  can  obtain  detailed  and  aggregated  resource 
and  cost  history  information  plus  probability  evaluation  of 
future  resource  and  funding  requirements.  The  reports  needed  to 
update  the  model  inputs  are  made  periodically,  as  decided  by  the 
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program  manager#  by  whatever  level  of  management  he  decides  pro¬ 
vides  him  with  necessary  detail;  therefore,  he  can  establish  a 
system  of  information  and  projection  which  allows  him  to  carry 
out  control  functions. 

Other  variations  of  outputs  concern  budget  information  pro¬ 
vided  by  various  network  analyzer  computer  programs.  For 
example,  VERT  produces  two  types  of  cost  data  for  each  selected 
decision  point  and  for  each  terminal  point.  The  first  type, 
called  "path  cost",  provides  the  cost  distribution  of  all  activi¬ 
ties  on  the  path  (sequence  of  activities)  which  led  to  that  node 
(during  each  run).  The  second  type,  called  "overall  cost",  pro¬ 
vides  not  only  the  path  cost  but  adds  to  it  the  cost  of  all  acti¬ 
vities  processed  prior  to  the  completion  of  the  decision  point 
and  provides  the  distribution  of  t_his  value.  Such  information 
can  be  collected  to  provide  overall  budget  estimates  [22].  These 
costs  are  illustrated  in  Figure  F-5  by  the  darkened  lines. 


Figure  F-5 

ACCUMULATIQN  OF  COST  DATA 


In  computer  programs  that  track  multiple  resources  (such  as 
TRANSIM  V),  the  budgetary  information  can,  in  principle,  be  at 
the  level  of  detail  of  appropriation  category  or  even  budget 
account  (however,  no  examples  of  these  levels  of  detail  have  been 
found  in  actual  use). 

3.  DECISION  ANALYTIC  METHODS 

a.  Introduction 

Decision  analysis  provides  a  framework  for  greater  depth  in 
risk  assessment  than  do  other  approaches.  A  cricital  person 
might  feel  that  many  of  the  existing  risk  assessments  will  not 
necessarily  assise  a  manager  in  making  a  decision.  The  manager 
may  know  more  facts  but  feel  that  he  has  no  more  insight  into  the 
effects  or  desirability  of  alternative  decisions.  The  more 
extensive  applications  using  decision  analysis  should  overcome 
chis  problem,  since  the  core  of  decision  analysis  is  the  deter¬ 
mination  of  a  decision  maker's  pattern  of  preferences  so  that  he 
can  be  logical  and  consistent  in  satisfying  those  preferences. 

While  the  advantages  of  decision  analysis  are  real,  a  manager 
cannot  just  tell  the  people  who  are  applying  a  network  model  ap¬ 
proach  to  change  approaches.  Of  the  literature  researched  for 
this  handbook,  the  only  examples  of  the  decision  analytic  ap¬ 
proach  found  were  those  of  a  small  group  of  U.S.  Army  analysts 
(12],  although  it  is  understood  that  Naval  War  College  Faculty 
personnel  have  used  this  method  also.  (The  principal  service 
methodological  works  are  [12],  and  [131,  and  [14].) 

In  the  application  of  this  method,  the  analyst's  approach  is 
more  like  a  comprehensive  systems  analysis  than  the  methods 
discussed  previously  in  this  appendix.  For  example,  the  author 
of  [13]  says: 

"As  a  basic  objective  of  risk  analysis  is  to  create  a 
quantitative  and  experimental  laboratory  to  study  program  suc¬ 
cess,  the  general  methodology  for  a  risk  analysis  is  quite  simi- 
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lar  to  the  steps  involved  in  systems  analysis,  systems  engineer¬ 
ing,  or  industrial  dynamics.  The  steps  include  the  following: 

1.  identify  objective 

2.  state  alternatives 

3.  collect  data 

4.  construct  model 

5.  simulate/apply  model 

6.  validate  model 

7.  obtain  criteria  and  trends" 

The  same  author  discusses  risk  analysis  objectives  in 
general: 

"It  is  important  to  ask  ourselves  whether  or  not  risk  analy¬ 
sis  contributes  to  acquisition  management.  A  risk  analysis 
should  identify  the  following  areas  to  the  acquisition  personnel: 

1.  potential  problems  areas 

2.  consequences  of  failure 

3.  low  risk  program  areas 

4.  requirements  versus  state-of-the-art  trade-offs 

5.  adequacy  of  acquisition  time 

6.  sufficiency  of  appropriations 

7.  optimum  allocation  of  funds 

8.  data  gaps/recommend  studies  and  concepts 

9.  sensitive/critical  parameters" 

It  is  clear  that  the  author  of  the  quoted  paragraphs  con¬ 
ceives  of  a  study  requiring  depth  of  investigations,  breadth  of 
knowledge,  and  significant  effort — a  study  which  contributes 
insight  into  many  aspects  of  the  program  under  assessment. 

b .  The  Model 

In  the  decision-analytic  approach,  the  program  model  is  for¬ 
mulated  by  reference  to  a  decision  tree  in  which  the  elements  are 
the  major  decision  points  in  program  development.  Although  the 
tree  itself  may  never  be  drawn,  all  relevant  events  must  be 
listed  and  an  analysis  made  to  determine  problems  that  can  occur 
during  each  phase  of  the  process  of  arrival  at  the  decision 
points.  Experts  are  consulted  to  identify  each  problem  and 
possible  problem  resolution  which  will  be  considered  (keeping  in 
mind  the  desirability  of  restricting  the  proliferation  of  the 
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tree),  and  to  assign  probabilities  to  the  various  problems  and 
resolutions.  Any  realistic  and  convenient  number  of  sequential 
resolution  efforts  can  be  postulated. 

Each  problem  is  defined  so  that  it  is  independent  of  the 
others;  that  is,  knowledge  of  its  occurrence/nonoccurrence 
does  not  alter  the  probability  of  occurrence  of  any  other. 

A  first  item  of  input  required  for  each  possible  problem 
resolution  is  its  cost.  The  reader  will  recognize  that  the  ele¬ 
ments  of  this  model  are  such  that  the  output  will  be  probabili¬ 
ties  of  success  or  failure  and  expected  costs.  The  inputs  can  be 
tabulated  as  in  Table  P-1,  where  PQ1  is  the  probability  of 
occurrence  of  problem  1,  PQ2  that  of  problem  2.  R^2  is  the  first 
resolution  of  problem  2,  R22  the  second  resolution  of  problem  2, 
and  PS21  is  the  probability  of  success  for  the  first  resolution 
of  problem  2,  Ps22  the  probability  of  success  for  the  second 
resolution  of  problem  2. 


Table  F-l 

DECISION  ANALYSES  PROBLEM/RESOLUTION  TABLE 

Potential  Probability  Possible  Probability  Cost 

Problems _ of  Occurrence _ Resolutions _ of  Success  Impact 


The  entries  of  cost  impacts  are  the  costs  required  to  obtain 
the  corresponding  resolution.  For  example,  C32  is  the  cost  of 
applying  resolution  2  to  problem  3. 


Having  determined  the  information  needed  for  this  kind  of 
table,  the  analyst  defines  the  events  which  are  the  combinations 


of  problem  occurrence  and  non-occurrence  and  their  resolution  and 
non-resolution . 

c .  Implementation 

To  implement  the  model,  the  analyst  enters  his  decision  tree 
information  as  a  network  into  any  available  probabilistic  network 
computer  program.  The  probabilities  he  has  formulated  will 
determine  the  cost  and  performance  outcomes  of  each  completion  of 
the  decision  sequences.  After  a  large  number  of  repetitions,  he 
can  take  the  outputs  collected  to  obt-in  values  permitting  the 
plotting  of  such  graphs  as  Figure  F-6 . 

d .  Outputs 

Outputs  are  exemplified  by  Figure  F-6.  The  analyst  can  see 
from  the  curve  labeled  "probability  of  occurrence",  for  example, 
the  cost  growth  corresponding  to  the  50  percent  probability  of 
occurrence  (which  would  provide  a  TRACE  under  this  analytical 
approach — in  the  figure,  about  $3  million). 

So  far  we  have  information  similar  to  that  produced  by  other 
risk  assessment  techniques  plus  that  from  having  the  "probability 
of  success"  curve  on  the  same  axis.  Perhaps  not  all  of  the  risk 
area  "identifications"  (as  recommended  in  the  passage  quoted  pre¬ 
viously  from  [14])  have  been  made,  but  the  approach  does  interre¬ 
late  performance,  cost  and  risk  in  a  form  amenable  to  trade¬ 
off.  ^  Note  that  this  analysis  shows  that  for  any  level  of  cost 
growth  (impact),  probability  of  program  success  does  not  exceed 
about  0.4. 

The  presentation  of  information  typified  by  Figure  F-6  is  in¬ 
sufficient  for  budgetary  purposes  as  shown;  that  is,  there  is  no 
appropriation  category  or  budget  year  breakdown.  There  is  no 


4[6j  also  describes  one  of  the  more  elaborate  network  models, 
RISNET. 
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Probability  ot  Occurrence 


COST  IMPACT  IN  t-MILLION 


COST 

. PROBABILITY 

CUMULATIVE 

PROBABILITY 

CUMULATIVE 

<K$) 

OCCURRENCE 

PROBABILITY 

SUCCESS 

PROBABILITY 

0 

0.000 

0.000 

0.000 

0.000 

724 

0.004 

0.004 

0.001 

0.001 

1,445 

0.022 

0.026 

0.014 

0.015 

2,172 

0.094 

0.120 

0.045 

0.060 

2,896 

0.202 

0.322 

0.086 

0.146 

3,620 

0.243 

0.565 

0.097 

0.243 

4,344 

0.200 

0.765 

0.076 

0.319 

5,068 

0.128 

0.893 

0.046 

0.365 

5,792 

0.066 

0.959 

0.019 

0.384 

6,516 

0.027 

0.986 

0.007 

0.391 

7,239 

0.010 

0.996 

0.002 

0.393 

7,963 

0.003 

0.999 

0.000 

0.393 

8,687 

0.000 

0.999 

0.000 

0.393 

Figure  F-6 

COST  IMPACT  DISTRIBUTION 

reason  in  principle,  however,  why  the  specified  model  phases  can¬ 
not  be  the  fiscal  years,  or  that  analysis  at  the  appropriation 
category  level  cannot  be  carried  out.  Since  no  analyst  has 
accomplished  this  yet,  extra  effort  would  be  needed  to  ensure 
adequacy  of  application. 
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The  outputs  available  from  another  decision  analytical 
approach  to  risk  assessment  (Probabilistic  Event  Analysis  -  PEA) 
include  those  shown  in  Figures  F-7  and  F-8  taken  from  [24]. 
These  figures,  which  are  really  combinations  of  worksheets  and 
results,  provide  a  summary  level  form  of  budget  risk  analysis. 5 

4.  THE  METHOD-OF-MOMENTS 

a .  Introduction 

The  method  of  moments  provides  a  way  to  determine  the  distri¬ 
bution  of  an  uncertain  cost  that  is  some  function  of  other  uncer¬ 
tain  costs.  The  function  may  involve  summation,  multiplication, 
exponentiation,  or  a  combination. 

Probability  theory  includes  various  theorems  for  determining 
the  probability  distribution  of  uncertain  values  which  are  sums 
or  products  of  other  uncertain  values.  However  in  many  cases, 
the  assumptions  that  must  be  made  to  keep  the  mathematics  tract¬ 
able  so  restrict  matters  that  applications  to  real  systems  are 
either  useless  or  impossible.  Previous  to  the  method-of -moments, 
the  complexity  of  project  element  interrelationships  has  made 
project  models  too  difficult  or  impossible  to  solve  mathemati¬ 
cally.  This  is  one  reason  why  the  technique  of  simulation  has 
gained  favor.  However,  now  that  the  long-neglected  method-of - 
moments  has  been  reexamined,  investigators  have  devised  mathema¬ 
tical  methods  for  solving  a  number  of  project  management  problems 
with  simple  computer  or  hand  calculator  routines.  The  theory 
behind  these  techniques  is  found  in  [20],  and  some  of  the  actual 
applications  are  presented  in  [26]  and  [27],  Discussions  of 
aspects  of  them  appear  below. 


5The  unique  aspect  of  the  decision  analytic  approach — the  use 
of  utility  theory — is  described  in  [9];  however,  no  applica¬ 
tion  of  that  theory  has  been  found  in  any  of  the  papers 
reviewed  for  this  handbook. 
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b.  The  Model 

To  use  the  method  of  moments,  the  first  item  needed  is  a 
mathematical  model.  If  cost  is  the  variable,  a  convenient  model 
is  a  Work  Breakdown  Structure  [21].  The  model  structure  repre¬ 
sents  the  analyst's  concept  of  the  objective  aspects  of  the  total 
model . 


Since  the  major  advantage  of  the  method  of  moments  over  the 
method  of  probabilistic  network  analysis  is  that  it  simplifies 
the  effort,  it  is  appropriate  to  keep  the  problem  concept  simple 
also.  The  model  defined  should  not  be  encyclopedic,  and  indeed, 
some  investigators  have  recommended  that  no  more  than  20  or  30 
elements  be  included  in  the  model,  although  as  many  as  63  are 
mentioned  in  [26].  On  the  other  hand,  the  more  the  model  ele¬ 
ments  are  aggregated,  the  less  likely  it  is  that  an  expert  can 
give  credible  estimates  of  uncertain  values.  The  analyst  must 
choose  a  compromise,  and  no  scientific  principle  tells  him  how  to 
do  so. 


c.  Implementation 

The  first  and  easiest  application  is  that  developed  in  [26]. 
The  application  allows  the  analyst  to  obtain  two  output  probabi¬ 
lity  curves.  These  represent  two  extremes,  that  of  complete 
statistical  independence  and  that  of  complete  dependence.  The 
two  bracket  an  unknown  exact  probability  curve  that  represents 
the  integration  of  all  the  uncertainty  information  in  an  additive 
cost  model.  This  is  necessary  because  only  the  extremes  of  com¬ 
plete  independence  of  all  component  values,  on  the  one  hand,  and 
worst-case  dependence  of  the  dependent  component  values,  on  the 
one  other  hand,  are  easily  solved.6 


6For  those  with  some  background  in  probability,  we  assume  that 
the  values  that  are  correlated  have  a  complete  positive 
correlation;  i.e.,  a  correlation  coefficient  equal  to  one. 
This  provides  a  "worst-case",  while  independence  provides  a 
"best-case" . 


F-22 


■v?i 


The  method  makes  use  of  another  simplifying  assumption:  that 
for  the  case  with  dependence,  a  normal  resultant  CDF  can  be  used 
based  on  a  PDF  with  the  mean  and  variance  determined  by  the 
mathematics  of  this  application.  Although  it  is  clear  that  the 
true  resultant  cannot  be  exactly  of  the  normal  form  (costs  cannot 
go  negative) ,  the  approximation  is  nonetheless  useful  when  there 
is  a  reasonably  large  number  of  input  uncertain  values  (greater 
than  25  seems  reasonable,  as  long  as  no  one  value  strongly  domi¬ 
nates  the  results) . 

As  an  example,  let  us  take  a  cost  model  devised  so  that  all 
elements  add  to  a  total,  and  none  has  multiplication  or  division 
of  one  uncertain  value  by  another  or  uses  any  in  an  exponential 
expression . 

Having  first  defined  the  model,  the  analyst's  next  task  will 
be  determination  of  quantitative  estimates  of  the  cost  element 
variables  and  of  the  uncertainty  surrounding  them.  (Do  not  forget 
that  uncertainty  is  a  subjective  property  of  the  expert,  not  an 
objective  property  of  the  costs.)  There  are  a  number  of  methods 
for  obtaining  the  quantitative  statements  of  uncertainty 
discussed  in  Appendix  D,  but  none  are  perfect.  This  part  of 
problem  model  formulation  represents  a  sensitive  area,  and  the 
program  manager  whc  is  interested  in  the  quality  of  a  risk 
assessment  will  be  concerned  to  ensure  that  efforts  have  been 
made  to  obtain  consistent  probability  assessments  reflecting  the 
experts'  opinions  as  accurately  as  possible. 

In  keeping  with  the  spirit  of  simplification,  it  is  possible 
to  consider  the  PDFs  describing  the  experts  uncertainty  to  be 
representable  by  only  two  types:  triangular,  as  in  Figure-  F-9, 
and  uniform,  as  in  Figure  F-10.  We  need  both  the  triangluar 
PDF  and  the  uniform  PDF  to  satisfy  both  the  expert  who  is 
willing  to  give  a  range  of  possible  values  (e.g.,  costs)  and  a 
most  likely  cost,  and  the  one  who  is  unwilling  to  say  that  any 
cost  is  more  likely  than  any  other  value  within  some  range. 
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Figure  F-9 
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Figure  F-10 
UNIFORM  PDF 


The  data  for  all  of  the  uncertain  costs  can  be  collected  on  a 
fo.m  similar  to  thar  of  Figure  F-ll  (modified  from  [27}. 
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Figure  F-ll 

DATA  COLLECTION/WORK  SHEET 

In  using  the  form  of  Figure  F-ll,  the  analyst  fills  in  the 

Input  section  with  the  names  of  the  cost  elements  and  the  values 

given  by  his  panel  of  experts  at  the  probability  levels  (P)  with 

which  they  are  confortable.  By  means  of  hand  calculator 

programs7  he  computes  the  results  to  the  right  of  each  element 

and  adds  them  in  accordance  with  the  following  rules: 

< 

(1)  Add  all  mean  values. 

(2)  Note  all  elements  having  a  dependency  on  each  other. 
Add  their  standard  deviations  and  square  them  to  obtain 
a  "variance"  of  their  sum.  Add  that  to  the  sum  of  the 
variances  of  all  other  elements. 

(3)  Find  the  square  root  of  the  value  resulting  from  step  2. 

(4)  Determine  the  cumulative  normal  distribution  for  the 
uncertain  value  having  the  total  mean  found  in  step  (1) 
and  the  variance  found  in  step  (3). 


7The  needed  programs  do  not  appear  in  any  of  the  references, 
but  writing  them  from  information  in  the  references  should  be 

relatively  simple. 
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Having  completed  one  set  of  outputs,  the  analyst  should  now 
repeat  the  computations,  but  this  time  add  all  variances  (no 
standard  deviations).  This  gives  the  output  for  a  model,  the 
inputs  of  which  are  all  independent.  The  two  curves  then  form 
bounds  on  the  "true"  output  distribution .8 


This  "true"  distribution  provides  the  information  needed 
either  to  understand  what  the  experts  have  been  saying  collec¬ 
tively  about  the  proabibility  of  not  exceeding  a  budget,  or  to 
calculate  a  budget  which  has  a  manager-selected  probability  of 
not  being  exceeded. 


Outputs 


Figure  F-12  exhibits  the  kind  of  simple  display  obtainable  by 
use  of  hand  calculator  programs.  The  display  is  printed  on  a 
tape,  which  accounts  .for  the  normal  CDF  being  presented  ver¬ 
tically.  The  graph  shows  the  analyst  that  he  needs  $59.44  in 
order  to  have  three  chances  out  of  four  of  not  exceeding  his 
budget. 


It  also  shows  that  the  mean  value  (the  same  as  the  most 
likely  and  the  50^  percentile  when  a  normal  CDF  is  used)  is 
$52.89;  that  the  standard  deviation  is  $9.72,  or  18  percent  of 
the  mean;  and  that  his  budget,  $45.00,  has  only  a  21  percent 
chance  of  not  being  exceeded.  The  maximum  and  minimum  amounts 
given  by  his  panel  of  experts  totalled  $76.00  and  $31.00,  respec¬ 
tively  . 


Of  course  an  additive  model  is  a  simplification.  Further¬ 
more,  results  of  network  simulations  show  that  interdependencies 
limit  the  use  of  a  normal  distribution  in  programs  of  any  great 
complexity.  Consequently  this  sort  of  a  model  should  be  regarded 
only  as  a  first  approximation,  to  be  refined  as  resources  permit. 


®The  unknown  distribution  is  "true"  in  the  sense  that  it  lies 
between  known  bounds.  It  cannot  actually  be  true  since  it  is 
a  normal  distribution,  which  we  chose  in  order  to  simplify 
the  problem. 
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Figure  F-12 

METHOD  OF  MOMENTS  HAND  CALCULATOR  OUTPUT 

The  authors  of  [26]  also  refine  this  approach  in  [27]  showing 
the  applications  for  multiplicative  and  other,  more  complex  func¬ 
tions  of  uncertfin  quantities.  They  then  use  the  moments  derived 
to  formulate  another  type  of  PDF  and  CDF  in  place  of  the  normal 
distribution  shown  herein.  This  other  type  of  PDF  possesses  the 
advantageous  features  of  being  flexible  in  shape,  the  shape  being 
dependent  on  the  values  of  parameters  used  in  their  formulas. 
The  moments  found  using  the  me thod-of -moments  allow  an  analyst  to 
define  the  parameters  for  this  type  distribution  so  that  it  will 
fit  the  mathematical  relationships  of  the  uncertain  inputs  to  the 
analyst's  model.  < 


Another  approach  to  the  problem  of  dependency  is  that  deve¬ 
loped  in  [31].  In  this  paper,  dependencies  are  assumed  to  stem 
from  an  underlying  but  unknown  factor,  so  that  the  observed  (or 
estimated)  uncertain  values  are  assumed  to  be  actually  different 


F-27 


functions  of  one  independent  uncertain  value  and  another ,  under¬ 
lying  one  common  to  all  the  seemingly  dependent  higher  level 
values . 

The  ambitious  analyst  may  wish  to  pursue  this  approach,  but 
must  be  careful  to  assure  himself  that  he  restricts  the  dependent 
uncertain  values  to  those  that  truly  appear  to  be  dependent. 

The  foregoing  paragraphs  demonstrate  an  approach  that  can 
assist  a  manager  of  a  small  program.  To  avoid  mathematically 
complex  explanations  in  this  handbook,  a  simplistic  approach  was 
used,  and  therefore  the  power  available  in  the  method  of  moments 
has  not  been  shown.  The  more  mathematically  adept  analyst 
wishing  to  exploit  the  method  more  fully  is  advised  to  read  [20], 
[26],  [27],  [30],  and  [31]. 

5.  WBS  SIMULATION  METHODS 

a.  Introduction 

This  method  stems  from  the  same  viewpoint  as  the  method-of- 
moments,  that  a  cost  estimate  is  composed  of  element  costs  which 
cannot  be  known  with  certainty.  A  cost  estimate  is  usually  shown 
as  a  sum  of  element  costs,  but  the  element  costs  themselves  may 
be  composed  of  subelement  costs  which  are  multiplied  together  or 
are  exponents,  or  have  some  other  relatively  complex  mathematical 
assocation.  As  was  stated  in  paragraph  4. a.,  this  has  made  the 
determination  of  the  PDF  of  total  cost  from  the  PDF's  of  the  ele¬ 
ment  c-nd  subelement  costs  so  difficult  as  to  be  a  practical 
impossiblity  until  recently.  This  situation  has  changed  bacause 
of  the  method-of-moments  and  the  availability  of  powerful,  rela¬ 
tively  inexpensive  computers.  The  WBS  simulation  technique  makes 
use  of  the  power  of  a  computer  to  repeat  a  complicated  set  of 
computations  a  great  many  times,  so  that  a  large  number  of  cost 
estimate  possibilities  can  be  collected  to  see  how  frequently 
they  each  occur.  From  the  frequencies  the  computer  can  form  a 
PDF. 
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b.  The  Model 


The  model  for  this  technique  is  the  basic  cost  breakdown  of 
the  program  cost  estimate.  Usually  this  consists  of  a  WBS  com¬ 
bining  engineering  and  functional  elements  for  which  costs  can  be 
estimated.  The  element  costs  must  (as  in  the  method-of -moments ) 
be  further  described  by  probability  statements  like  PDF's.  With 
some  of  the  methods  ([5],  [6])  it  is  necessary  to  define  the  ele¬ 
ments  in  such  a  way  as  to  ensure  their  statistical  independence 
(see  Appendix  E)  while  in  others  [15]  dependence  is  conceptually 
included.  In  some  of  the  methods,  the  model  includes  a  few  con¬ 
ceptual  sources  of  possible  error  in  the  estimates,  and  a  com¬ 
puter  routine  picks  the  source  considered  to  override  the  others, 
and  thus  selects  the  PDF  associated  with  that  source. 

c .  Implementation 

Consideration  of  model  size  applies  here  as  in  other 
techniques;  it  is  helpful  to  keep  the  number  of  cost  elements 
down  to  a  convenient  size — perhaps  sixty  or  less.  This  will  per¬ 
mit  both  detailed  data  gathering  and  also  visibility  of  major 
effects . 

In  some  of  the  WBS  simulation  methods,  the  analyst  or  subject 
matter  experts  who  provide  cost  estimates  and  ranges  also  choose 
PDF's  by  shape  from  a  selection  provided  by  the  analyst.  After 
the  analyst  has  checked  to  ensure  consistency,  he  enters  all  the 
costs  and  PDF's  into  the  computer.  The  computer  produces  results 
based  upon  assumptions  of  dependence  and  independence,  and  cer¬ 
tain  extremes  of  input  PDF  shape  (which  is  related  to 
uncertainty).  This  range  of  outputs  allows  the  analyst  to  see 
the  limits  resulting  from  assumptions  regarding  dependence  and 
relative  uncertainty. 

In  [5],  selection  of  the  cost  element  PDF  is  made  by  the  com¬ 
puter  after  the  analyst  has  described  four  sources  of  cost  esti¬ 
mating  error  (uncertainty).  These  sources  must  be  described  as 
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independent  of  one  another.  The  four  sources  are:  estimating 
technique  or  its  method  of  application;  program  schedule;  tech¬ 
nology  advancement;  and  system  configuration.  The  analyst  inputs 
four  quantities,  the  most-likely  cost  of  the  element  (normally 
the  single  point  cost  estimate)  and  three  other  numbers  for  each 
error  source  as  it  affects  the  element.  These  three  numbers  are 
the  highest  and  lowest  cost  that  the  error  source  could  cause, 
and  a  measure  of  uncertainty.  The  uncertainty  numbers  are 
derived  from  the  verbal  descriptions,  "low",  "medium  low", 
"medium  high",  and  "high",  by  reference  to  typical  PDF's  that  can 
be  described  as  showing  those  uncertainty  ranges. 


d .  Outputs 

The  various  computer  programs  developing  this  basic  technique 
output  somewhat  different  information,  but  the  three  reviewed  for 
this  handbook  do  have  certain  similarities.  All  three  include 
tables  of  probabilities  for  each  cost  increment  selected  and  pro¬ 
babilities  that  total  cost  will  not  exceed  given  dollar  amounts. 
This  is  essentially  the  information  contained  in  a  PDF  and  a  CDF, 
and  two  of  the  programs  [5]  [6]  plot  the  resulting  CDF's  and 

print  various  statistics  like  mode  (most  likely)  and  mean 
(average)  and  costs  corresponding  to  selected  probabilities.  One 
of  these  computer  programs  [5]  also  plots  the  input  PDFs  for  exa¬ 
mination  and  another  [15]  prints  the  CDF  information  resulting 
from  assuming  independence  or  dependence  plus  that  resulting  from 
extreme  assumptions  of  uncertainty. 

G  .  THE  GRAPHIC  METHOD 

a .  Introduction 

The  preceding  sections  of  this  Appendix  have  included  one  ex¬ 
position  of  a  summary  level  application  (the  method-of -moments ) . 
This  method  was  described  so  that  managers  of  programs  enjoying 
lower  levels  of  support  might  be  able  to  accomplish  some  level  of 
cost  risk  assessment.  This  section  provides  another  summary 
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level  method  for  those  program  managers  who  cannot  obtain  support 
for  more  searching  assessments.  The  graphic  method  is  the  pro¬ 
duct  of  the  author  of  [17],  and  his  paper  recounts  the  need  for 
summary  method  approaches.  As  he  says. 


"The  effort  expended  and  care  taken  in  obtaining  risk  data, 
plus  the  cost  of  developing  and  running  a  computer  model,  tends 
to  limit  the  use  of  risk  assessment  to  major  programs  or  to 
larger  companies  where  sufficient  resources  are  available. 
Therefore,  in  order  to  broaden  the  application  of  risk  assess¬ 
ment,  an  alternate  and  simpler  technique  was  developed.  The 
technique  provides  a  viable  risk  assessment  that  is  independent 
of  the  computer,  although  it  requires  some  constraints  on  the 
flexibility,  accuracy  and  usefulness  of  the  resultant  analysis. 
The  computer  simulation  model  is  replaced  by  a  'manual  model'.  A 
pencil  is  used  instead  of  the  keypunch  and  computations  on  a  hand 
held  calculator  replace  the  computer  itself.  The  manual  method, 
while  not  as  comprehensive  as  the  computerized  one,  provides  a 
rapid  and  less  costly  mea,is  of  preparing  small  analyses ....  this 
technique  can  serve  as  an  important  first  step  which  may  pave  the 
way  for  future  use  of  a  more  sophisticated  computer  model 
methodology . " 

A  weakness  of  the  method  described  is  that  it  produces  a  set 
of  probability  density  functions,  as  descriptions  of  expert 
uncertainty,  which  are  intuitively  unsatisfying  (see  Figure 
F-13)  . 


COST 

Figure  F-13 

EQUIVALENT  DENSITY  FUNCTION  FOR  GRAPHIC  METHOD 


The  strength  is  that  the  PDF's  do  contain  a  certain  intuitively 
desirable  feature,  skewness,  and  they  are  easy  to  work  with. 


b.  T'.ie  Model 


Like  other  methods,  application  of  this  method  starts  with 
the  establishment  of  an  additive  cost  model.  As  with  the  simpli¬ 
fied  method  of  moments,  the  analyst  must  define  cost  elements 
that  are  statistically  independent,  and  he  will  want  to  restrict 
the  number  of  elements  to  a  size  that  he  can  handle  adequately; 


i  .e. ,  20  to  30 


With  his  model  and  the  list  of  cost  elements  as  a  basis,  the 
analyst  next  obtains  expert  estimates  of  costs  and  cost  ranges. 
In  this  approach  it  is  necessary  to  obtain  a  50th  percentile  cost 
as  the  central  estimate;  the  experts  are  asked  to  estimate  the 
cost  for  which  the  element  has  as  great  a  likelihood  of  overrun 
as  underrun.  The  expert  then  estimates  two  extreme  cost  values 
and  their  associated  probabilities.  These  three  values  should 
represent  the  50th  percentile  and  2  extreme  points  having  equal 
probability  differences  with  50  percent  (e.g.,  the  5th  and  95th 
percentile) .  This  method  assumes  that  the  higher  should  be 
farther  from  the  central  value  than  the  lower.  In  other  words, 
the  distribution  function  should  have  a  righthand  skew.  Many 
risk  and  cost  analysts  find  righthand  skewness  desirable  because 
it  estimates  that  costs  are  more  likely  to  overrun  than  underrun. 
Whereas  in  some  other  methods  the  right  skewed  distribution  is 
modeled  as  a  triangle  or  a  well-known  skewed  curve,  in  this 
method  curves  are  used  that  are  halves  of  normal  distributions. 


The  unusual  feature  of  the  graphic  method  is  its  ease  of 
deriving  a  skewed  curve  to  represent  an  individual's  uncertainty 
in  estimating  a  cost.  The  method  is  easy  because  the  analyst  can 
make  use  of  graph  paper  designed  so  that  a  straight  line  on  it 
represents  a  cumulative  normal  distribution  function,  and  a  user 
has  only  to  plot  two  points  or  a  point  and  a  line  to  obtain  such 


F-32 


J 


7W^ 


a  distribution.  Figure  F-14  shows  a  plot  of  two  lines  for  the 
three  points  estimated  by  the  expert.  Figure  F-13  shows  what  the 
two-line  plot  has  represented  in  terms  of  normal  density  func¬ 
tions  . 


H0IMIUTY 

Figure  F-14 


ELEMENT  PROBABILITY  CURVE 

The  solid  (discontinuous)  curves  serve  as  the  three-point 
density  function,  where  the  left  side  is  the  left  side  of  a  low 
variance  normal  density  function  and  the  right  side  is  that  of 
one  having  a  relatively  high  variance.  Since  normal  density 
functions  are  symmetric,  there  is  an  area  of  0.50  under  each  half 
curve;  thus  the  three-point  function  integrates  to  the  value  1. 

This  synthetic  curve  looks  like  no  standard,  defined  probabi¬ 
lity  density  curve.  It  is  discontinuous,  and  it  extends  to  nega¬ 
tive  numbers,  however  it  is^  easy  to  manipulate.  It  also  has  the 
desirable  feature  of  being  skewed  to  the  right.  It  should  be 
noted  that  although  the  density  function  is  discontinuous,  the 
cumulative  distribution  function  (see  Figure  F-15)  is  not.  In 
other  words,  the  expert  can  still  say  meaningful  things  about  the 
chances  of  costs  being  less-than-or-equal-to  any  number  covered 
by  the  curve. 
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Figure  F-15 

CDF  FOR  GRAPHIC  METHOD 

c .  Implementation 

This  summary  method  gives  a  way  to  combine  the  uncertain  cost 
element  distributions  for  cost  elements  which  are  to  be  added. 
The  method  has  not  been  justified  by  theory,  but  the  originator 
inferred  it  from  the  results  of  simulations,  and  the  inferrence 
from  those  results  of  a  few  easily  applied  algebraic  rules.  The 
rules,  taken  from  [17],  are  presented  below.  Start  with  the 
following  definitions: 

o  Vio,  V50,  V90  -  the  values  plotted  as  the  10th,  50th,  and 
90th  percentile  for  a  given  cost  element.  (Vio»  v90r  and 
V ^90  may  not  be  the  expert-supplied  values  but  derived 
from  the  normal  paper  plot). 

o  Skew  -  the  percent  deviation  from  a  straight-line  exten¬ 
sion  of  the  left  hand  half-line  at  the  90th  percentile. 
That  is,  it  is  the  difference  between  V90  and  a  value 
found  by  extending  the  left  half-line  (as  in  Figure  F-14). 
Note  that : 

v  90  =  2  V50  -  Vio- 
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Figure  F-16 

DETAIL  OF  ELEMENT  CURVE 

To  determine  values  for  finding  a  cost  distribution  which  is 
that  of  a  sum  of  the  element  costs,  for  each  cost  element: 

(1)  Add  all  ViO/  V50,  and  v90  values,  respectively. 

(2)  Find  skew  for  the  new  points  =  (V9q/V^9q)  _  j. 

(3)  Find  a  factor,  FA,  as  follows: 

FA  =  0.4  x  (skew) 

(4)  Find  a  value  for  a  factor,  N,  related  to  the  number 
of  cost  elements,  from  the  following: 


Number  of  Elements 
2  to  4 
5  to  10 
11  to  19 
20  to  30 

(5)  Find  another  factor,  FB 


N  Value 
0.6 
0.7 
0.8 
0.9 


FB  =  N  X  FA 

(6)  Adjust  the  V^Qr  V50,  and  V90  sums  to  obtain 
resultant  values,  denoted  RVio»  RV50r  an<*  RV90  as 
follows : 

RV10  =  (1  -  FB)  V10 
rV5q  =  (1  +  FB)  V50 
RV90  =  (1  +  0.5  FB)  V90 

(7)  Calculate  the  mean  of  the  resultant  distribution  in 
three  steps: 
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o  First,  find  the  skew  of  the  resultant,  Rskew. 


Rskew  =  (RVgo/RV^Q )  _  i.q 
o  Then  find  the  FA  factor  for  the  resultant,  RFA> 
RFA  =  0.4  x  (R  skew) 

o  Last,  find  the  expected  value  (mean),  REV. 

REV  =  (1  +  RFa)  RV50 

(8)  Complete  the  evaluation  by  plotting  the  new  values 
and  the  cost  levels  of  interest,  as  in  Figure  F-17. 

d .  Outputs 

Figure  F-17  shows  two  "curves"  and  an  estimate.  The  dashed 
curve  represents  the  result  of  summing  the  original  V^q,  V50,  and 
V90  values.  The  solid  curve  represents  the  adjusted  values  found 
with  RViO'  SV50,  and  RV90. 


Figure  F-17 

RESULTANT  AND  SUMMATION  PROBABILITY  CURVES 


The  horizontal  dashed  line  in  Figure  F-17  is  the  cost  level  of 
interest,  which  could,  for  example,  be  a  budget.  The  method's 
originator  comments  as  follows: 

"Note  that  the  original  summation  curve  has  a  50  percent 
probability  of  being  $4  million  less  than  the  estimate.  However, 
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after  transformation  to  the  resultant  probability  curve,  the 
value  at  50  percent  probability  is  $9  million  above  the  esti¬ 
mates.  The  probability  of  achieving  the  estimate  has  dropped  to 
40  percent  and  the  value  at  90  percent  probability  is  $237 
million  or  41  percent  potential  overrun  of  the  estimate." 

7 .  ESTIMATING  RELATIONSHIP  METHODS 

a .  Introduction 

The  Estimating  Relationship  methods  derive  their  source  from 
the  observation  that  costs  of  systems  seem  to  correlate  with  per¬ 
formance  variables.  These  independent  variables,  often  called 
explanatory  variables,  permit  the  use  of  regression  analysis  to 
describe  an  assumed  or  hypothesized  underlying  mechanism  relating 
the  variables  and  cost.  That  is,  cost  as  a  dependent  variable  is 
assumed  to  be  a  function  of  the  variables  plus  an  error  term. 
The  Cost  Estimating  Relationship  (CER)  approach  to  cost  esti¬ 
mating,  also  called  parametric  cost  estimating,  is  widely 
accepted  and,  even  for  complex  functions,  easy  to  apply.  This 
ease  of  application  makes  it  natural  to  attempt  the  use  of 
regression  techniques  and  estimating  relationships  in  risk  analy¬ 
sis.  The  approach  is  to  attempt  to  discover  acquisition  program 
characteristics  to  which  numbers  can  ">e  assigned  as  values  of 
explanatory  variables,  which  can  then  be  correlated  with  histori¬ 
cally  demonstrated  need  for  management  reserve.  The  user  of  this 
method  performs  a  regression  analysis  using  "actual"  management 
reserve  as  a  dependent  variable  to  determine  a  function  of  the 
explanatory  variables  so  that  he  can  estimate  management  reserve 
requirements  for  programs  not  in  his  data  base. 

A  basic  difference  between  the  use  of  such  a  parametric 
approach  for  cost  estimating  and  for  risk  analysis  is  that  the 
explanatory  variables  selected  in  the  risk  analysis  application 
surveyed  for  this  handbook  do  not  represent  performance,  or  out¬ 
put,  variables.  Therefore,  the  hypothesized  mechanism  described 
by  the  derived  function  of  explanatory  variables  has  less 
intuitive  justification. 
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A  danger  inherent  in  the  use  of  this  method  is  the  failure  to 
understand  the  relative  relationship  of  the  explanatory  variables 
to  the  dependent  variable.  The  selection  of  different  ranges  for 
each  explanatory  variable  in  itself  modifies  the  relative  effect 
and  when,  as  in  one  application,  the  values  of  the  separate 
explanatory  variables  are  simply  added  to  form  a  "rating,"  hidden 
assumptions  are  made  which  should  be  explicitly  justified  since 
they  are  easily  questionable.  The  application  of  the  more 
sophisticated  regression  technique  known  as  stepwise  regression 
might  obtain  quite  different  results.  Despite  these  concerns, 
the  technique  has  received  acceptance. 

b.  The  Model 

The  application  of  this  technique  is  described  in  [8].  Four 
contract  characteristics  which  exhibit  degrees  of  uncertainty  are 
evaluated  by  program  management  personnel.  Each  characteristic 
is  assigned  a  number  on  a  different  scale  for  each  characteris¬ 
tic.  The  four  characteristics  used  are  Engineering  Complexity 
(zero  to  five).  Contractor  Proficiency/Experience  (zero  to 
three).  Degree  of  System  Def initization  (zero  to  three),  and 
Multiple  Users  (zero  or  one).  The  sum  of  these  numerics  is 
entered  as  the  value  X  in 

y  =  0.192  -  0.037  X+  .009  X2 

This  formula  finds  the  percentage  management  reserve  require¬ 
ment,  y.  The  model  is  said  to  be  usable  only  for  X  values  be¬ 
tween  2  and  10.  Lower  values  indicate  lack  of  need  for 
management  reserve.  Experience  on  which  the  estimating  rela¬ 
tionship  is  based  indicates  that  programs  with  £  or  1()  ratings 
(management  reserve  over  50  percent)  were  given  frequent  enough 
review  to  ensure  adequate  funding. 

c.  Implementation 

The  values  assigned  to  each  of  the  four  program  characteris¬ 
tics  are  determined  by  using  an  existing  definition  for  each 
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level  or  value.  To  that  degree,  a  certain  amount  of  standar¬ 
dization  is  maintained.  Presumably,  the  same  person  evaluating  a 
number  of  different  programs  could  feel  confident  in  the  con¬ 
sistency  of  his  evaluations  and  resultant  management  reserves. 
On  the  other  hand,  it  is  not  clear  that  different  management 
offices  would  evaluate  a  common  list  of  programs  to  arrive  at 
similar  area  levels  or  summations.  A  question  needing  settlement 
before  this  method  can  be  used  with  confidence  is  one  of  relative 
scaling  among  characteristics.  The  assignment  of  a  "three" 
instead  of  a  "one"  to  a  program  characteristic  variable  has  the 
same  effect  for  each,  yet  it  is  not  clear  that  such  should  be  the 
case . 

d .  Outputs 

The  output  of  this  form  of  risk  analysis  is  a  percentage 
management  reserve.  According  to  [8],  the  historic  management 
reserves  used  consisted  of  contract  cost  growths — defined  as  per¬ 
centage  above  initial  contract  cost — to  accommodate  for  unknown 
program  events  and  for  alteration  of  the  system's  physical  or 
functional  characteristics.  Since  the  data  were  collected  on  the 
contract  level,  it  seems  that  such  validity  as  this  method 
possesses  relates  to  the  restricted  level  of  contract  management 
reserve . 

8.  THE  RISK  FACTOR  METHOD 


a .  The  Model 

For  this  method,  an  engineering  form  WBS  is  preferable  from 
the  point  of  view  of  obtaining  expert  assessment  of  the  WBS  ele¬ 
ment  cost  growth  possibilities.  This  method  is  more  intuitive 
than  the  other  methods  discussed  earlier;  less  demand  is  made  on 
the  expert  who,  in  practice,  is  often  a  cost  analyst  providing  an 
assessment  for  a  broad  range  of  activities  and  technologies.  No 
consideration  of  the  entire  range  of  possible  costs  is  requested, 
and  thus  a  bias  may  be  built  in.  There  can  also  be  a  tendency  to 
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give  less  care  in  supplying  the  cost  factor  than  is  used  in  | 
supplying  the  difficult  assessment  needed  for  other  methods.  In 
addition,  factors  are  by  definition  multiplicative;  that  is,  they 
will  be  used  to  obtain  upper  limits  of  cost  by  applying  the  fac¬ 
tors  to  the  WBS  cost  element  point  estimates.  Few  analysts  will 
claim  that  they  can  supply  the  factors  used  in  this  method  to  an 
accuracy  of  more  than  two  decimal  places,  therefore,  the  preci¬ 
sion  of  these  statements  is  necessarily  low,  especially  with 
highly  aggregated  or  high-cost  WBS  elements.  Furthermore,  little 
insight  is  available  through  this  type  of  information  for  deter¬ 
mining  critical  program  areas.  Despite  these  concerns,  the  tech¬ 
nique  is  widely  accepted. 

b .  Implementation 

The  primary  source  of  this  method  is  [18], 

A  Work  Breakdown  Structure,  described  in  [21],  is  an  outline  1 

?• 

form  or  hierarchy  of  engineering  tasks  or  budget  categories  for  ] 
describing  a  program.  \ 

In  the  use  of  this  method,  each  Work  Breakdown  Structure  ; 

(WBS)  element  identified  in  a  material  development  program  is  | 

analyzed  to  determine  the  specific  subelements  that  will  contri¬ 
bute  to  uncertainty  in  the  establishment  of  the  cost  of  execution 
of  that  element.  A  risk  factor  is  then  assigned  for  each  iden¬ 
tified  WBS  element  as  a  number  increased  over  1  by  an  amount  * 

representing  an  accumulation  of  estimated  proportional  increases  - 

in  the  cost  of  that  element  as  a  result  of  probable  changes  to  it 

I 

during  its  development.  These  increases  are  determined  by  a  ; 

judgement  based  on  the  analyst's  expectation  of  cost  growth.  1 

If  the  specific  program's  work  has  not  yet  been  identified  in  jj 

sufficient  detail,  risk  factors  are  assigned  on  the  basis  of  | 

larger  aggregates  of  work,  system  components,  or  system  types,  f 

depending  upon  the  detail  available.  Specific  considerations  to  1 

1 

be  included  in  this  judgement  are:  | 


4 

s 
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(1)  Requirement  to  develop  an  item  not  possible  with  existing 
technology. 

(2)  Requirement  to  develop  a  completely  new  item. 

(3)  Whether  major  integration  effort  will  be  necessary. 

The  following  paragraphs  supply  an  example  from  [18]. 

Assume  that  the  development  of  a  single  channel,  selectable 
frequency,  PM  VHF  radio  transceiver  is  desired,  as  in  Figure  12, 
page  IV-21. 

1.  Prepare  an  engineering  cost  estimate  (as  in  Table  [ F— 2 ] 
for  each  element  in  the  WBS. 

2.  Compute  a  risk  factor  for  each  WBS  element. 

3.  Multiply  the  risk  factor  by  the  element  cost  (enter  the 
results  as  column  (c)  of  Table  [ F— 2 ] ) . 


Table  F-2 

COMPUTATIONAL  METHODOLOGY 

a  be 


FM  VHF  Radio 

Engineering 

Risk 

Revised 

WBS  Elements 

Cost  Est . ,$ 

Factor 

Estimate 

Packaging 

21,000 

1.34 

28,140 

Tansmitter 

18,000 

1,13 

20,340 

Receiver 

12,000 

1.04 

12,480 

Power  Supply 

12,000 

1.20 

14,400 

Synthesizer 

25,000 

1.35 

33,750 

(Freq  Mult  Reduc) 

(10,000) 

1.02 

(10,200) 

(Oscillator ) 

( 15,000) 

1.57 

(23,550) 

TOTAL 

$88,000 

$109,110 

In  the  example,  risk  factors  are  developed  for  each  com¬ 
ponent  using  the  following  reasoning . 

1.  Examining  individual  components  results  in  the  conclusion 
that,  except  for  the  oscillator,  engineering  cost  estimates 


( considering  only  the  component  by  itself,  or  internal  effects) 
are  certain.  The  design  effort  is  nou  significantly  different 
from  previously  accomplished  work.  To  determine  the  contribution 
to  the  risk  factor  for  the  oscillator  due  to  internal  effects, 
costs  for  comparable  oscillators  from  three  other  systems  are 
examined . 

2.  The  design  process  is  highly  complex  and  iterative, 
often  causing  interaction  effects  among  system  components. 
Design  changes  to  the  oscillator  can  induce  design  changes  in  the 
power  supply  and  packaging  areas  which  in  turn  could  have  design 
impact  on  the  oscillator  again.  For  each  component  the  contribu¬ 
tion  to  the  risk  factor  due  to  external  effects  is  assigned 
(taking  into  account  all  potential  external  cost  contributors, 
such  as  interaction  and  funding  delays,  and  the  probabilities  of 
occurrence  of  these  costs). 

"The  risk  factor  for  each  component  is  a  composite  factor 
determined  by  combining  the  contributions  of  both  internal  and 
external  effects.  For  example,  it  is  estimated  that  the  total  of 
internal  and  external  forces  on  the  oscillator  will  result  in  a 
57%  increase  over  the  engineering  cost  estimate.  Therefore,  the 
risk  factor  is  1.57  (column,  b,  Table  [ F— 2 ] )  .  The  other  risk  fac¬ 
tors  are  similarly  general."  [18] 

c.  Outputs 

The  example  continues: 

The  program  risk  assessment  cost  can  now  be  compiled  as 
illustrated  in  Table  I F— 23 • 

Subsequently,  program  work  is  broken  out  by  year  for  program¬ 
ming  and  budgeting  purposes  using  the  original  engineering  esti¬ 
mate,  tne  risk  factors  and  revised  estimates,  as  shown  in  Table 
F-3.  The  unparenthesized  number  is  the  year's  original  element 
estimate  and  the  parenthesized  number  is  the  product  of  the 
unparenthesized  one  and  the  appropriate  risk  factor. 
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Table  [F-3] 


Annual  Cost  and  Risk  Cost 


WBS  ELEMENT 


YEAR  1 


ESTIMATED  COSTS  ($K) 
YEAR  2  YEAR  3  YEAR  4 


TOTAL 


Packaging 

3( 

4.02) 

4( 

5. 

.  36 ) 

6( 

8. 

,04) 

8' 

10 

.72) 

Transmitter 

6( 

6.78) 

6  ( 

6. 

,78) 

3  ( 

3. 

,39) 

3( 

3. 

39) 

Receiver 

4  ( 

4.16) 

4( 

4. 

.16) 

2( 

2. 

,08) 

2( 

2. 

08) 

Power  Supply 

4( 

4.80) 

4( 

4. 

,80) 

2  ( 

2. 

,40) 

2( 

2. 

40) 

Synthesizer 

10( 

14.05) 

6  ( 

7. 

,77) 

5  ( 

6. 

,75) 

4  ( 

5. 

18) 

21(28.14) 

18(20.34) 

12(12.48) 

12(14.40) 

25(33.75) 


TOTALS 


27(33.81)  24(28.87) 


18(22.66)  19(23.77) 


88(109.11) 


(NOTE:  Revised  estimate  figures  are  adjacent  to  the  engi¬ 
neering  estimate  for  each  year.) 

On  the  basis  of  the  above  calculations,  the  budget  request 
for  year  1  would  be  $33.3K,  and  the  programming  amounts  for  the 
subsequent  outyears  would  be  $28.87K,  $22.66K,  and  $23.77K, 
respectively,  for  a  program  total  of  $109. 11K. 


The  sole  applicability  of  risk  factors  assessment  is  for  cost 
estimating  and  budgeting.  The  type  and  quality  of  information 
derived  is  unsuitable  for  other  uses. 
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APPENDIX  G 


PRESENTATION  OF  RESULTS 

1.  GENERAL 

As  a  decision  influencing  document,  a  risk  assessment  can 
have  many  uses.  It  may  be  a  reference  for  subsidiary  decisions, 
for  reevaluations  as  time  passes,  and  a  basis  for  management  stu¬ 
dies  that  will  be  undertaken  in  the  future.  Assessments  that 
substantiate  major  decisions,  such  as  the  Decision  Risk  Analyses 
used  at  major  milestones,  and  those  that  provide  the  basis  for 
budget  submissions  will  be  under  increasingly  critical  review  as 
more  and  higher  levels  of  management  become  familiar  with  risk 
analysis  tools.  Managers,  therefore,  should  be  interested  in 
appropriately  rigorous  documentation  of  risk  assessments. 

Since  a  risk  assessment  is  an  effort  to  apply  scientific 
methods  to  objective  and  subjective  information,  the  framework  in 
which  the  assessment  is  described  should  enhance  and  enforce  the 
logic  of  scientific  inquiry.  The  method  of  scientific  inquiry  in 
matters  relating  to  management  is  often  termed  systems  analysis 
or  operations  research.  Thus  the  general  approaches  and  organi¬ 
zation  of  a  systems  analysis  or  operations  research  report  are 
appropriate  to  a  risk  assessment. 

2 .  REPORT  INTRODUCTION 


Each  report  should  begin  with  an  introduction  and  synopsis  of 
the  sections  to  follow.  The  first  item  should  identify  the  docu¬ 
ment  requiring  the  assessment.  Essentials  of  problem  formula¬ 
tion,  key  decisions  to  be  made,  alternatives  to  be  considered,  a 
brief  description  of  analytical  approach,  findings,  and  major 
recommendations  should  appear.  This  section  may  be  entitled 
Executive  Summary  and  should  serve  the  purpose  of  briefly  commu¬ 
nicating  enough  information  for  an  executive  to  understand  the 
issues,  have  confidence  in  the  analysis,  identify  the  conclu¬ 
sions,  and  determine  the  decision  implications. 
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3 .  PROBLEM  FORMULATION 


This  section  should  describe  the  decision  for  which  the 
assessment  is  being  performed.  The  section  should  describe  the 
program  in  summary  fashion  and  provide  details  relative  to  the 
decision  to  be  made.  If  the  decision  is  selection  of  one  or  more 
technical  alternatives,  this  section  should  describe  the  charac¬ 
teristics  they  incorporate — both  those  that  provide  orientation 
and  those  that  relate  to  the  decision. 

4.  APPROACH 

The  second  section  of  the  main  report  should  describe  in 
moderate  detail  the  general  methodology  to  be  used  in  the  analy¬ 
sis.  An  assessment  is  not  completed  by  the  exercise  of  a  piece 
of  software,  yet  most  network  analyses  depend  heavily  on  some 
piece  of  software.  The  software  should  be  completely  identified 
by  name  and  version  number.  Any  unusual  updates  or  features 
employed  should  be  described.  The  methods  of  representing  uncer¬ 
tainty  that  are  actually  in  the.  model — if  a  model  limitation-- 
should  be  explained  and  the  implications  of  the  limitation 
discussed.  If  the  software  can  accommodate  more  than  one  method 
of  representation  (e.g.,  PDF  or  explicit  list  of  probabilities, 
or  various  different  PDF's),  then  the  reasons  for  selection  of 
the  utilized  method  and  some  discussion  of  pros  and  cons  should 
be  included.  For  networks,  the  number  of  activities  (arcs)  and 
decision  points  (nodes)  should  be  given.  Problems  in  modeling 
and  compromises  for  purposes  of  simplifications  of  modeling 
should  be  discussed. 

a.  Assumptions .  Major  assumptions  should  be  identified. 
This  is  easy  for  some  assumptions,  but  frequently  an  analyst  is 
unaware  of  assumptions.  A  concentrated  effort  should  be  made 
during  the  assessment  process  so  that  all  assumptions  may  be 
identified  and  addressed. 


b.  Data  Collection.  A  subsection  should  be  devoted  to 
describing  the  categories  of  information  needed  for  the  analysis, 
the  organizational  sources  of  the  information,  and  the  techniques 
used  to  obtain  it.  The  techniques  could  include  any  of  the  face- 
to-face  techniques  described  in  Appendix  D  or,  for  example, 
questionnaires.  If  simplifications  to  data  are  made  they  should 
be  discussed  as  well  as  the  implications  of  the  simplifica¬ 
tions.  1  An  Appendix  should  list  the  data  and  the  data  sources 
by  name,  and  rationales  for  the  data  where  appropriate  (i.e., 
where  it  represents  subjective  judgment.) 

c.  Evaluation.  The  output  of  the  model  or  analysis  must  be 
described  in  sufficient  detail  so  that  the  analytical  approach  to 
its  evaluation  can  be  made  clear.  There  snould  be  a  description 
of  what  the  possible  combinations  and  ranges  of  outputs  can  imply 
to  the  program  manager. 

d.  Summary .  The  discussion  of  the  Approach  can  be  the  same 
in  many  different  assessment  reports  using  the  same  techniques, 

r, 

but  it  should  still  be  included  for  purposes  of  report  complete¬ 
ness  . 

5.  FINDINGS 

The  findings  are  factual  statements  such  as,  "The  analysis 
shows  that  there  will  be  a  P  percent  probability  of  funds  in 
excess  of  the  budget  being  needed  to  be  ready  for  milestone  X  on 
schedule",  or,  "The  analysis  shows  that  decision  point  Y  has  the 
highest  criticality  index  of  all  decision  points.  This  is 
because  activities  P,  Q,  and  R,  all  having  high  uncertainty, 
require  successful  development  of  component  J" .  Some  findings, 
while  not  major,  deserve  discussion  in  that  they  emphatically 
confirm  or  contradict  conventional  wisdom. 


^For  example,  one  study  in  the  bibliography  uses  most  likely 
values  (taken  from  a  baseline  cost  estimate)  and  derives  the 
extreme  values  by  applying  the  same  standard  multipliers 
against  all  most  likely  values.  The  PDF  form  used  was  the 
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a.  Sensitivity  Analysis.  A  sensitivity  analysis  determines 
the  effects  of  assumptions  on  the  findings.  Thus,  the  more 
searchingly  identified  the  assumptions,  the  more  clarifying  are 
the  sensitivity  analysis  and  the  more  convincing  are  the 
assessment's  findings  and  recommendations.  It  is  important  that 
assumptions  be  identified  early  in  the  assessment  process,  since 
reformulation  of  the  model  to  analyze  extreme  values  of  an 
assumption  may  cause  major  disruption.  Not  only  are  the  effects 
of  selected  counter-assumptions  important,  but  perhaps  even  more 
important  are  the  boundaries  in  the  regions  of  assumptions  at 
which  major  findings  and  major  recommendations  change.  For 
example,  although  a  manager  may  not  be  able  to  give  precise 
values  for  numeric  assumptions,  he  will  want  to  know  whether  they 
lie  well  within  a  region  in  which  decisions  do  not  change. 

b.  Recommendations .  A  proper  systems  analysis  or  opera¬ 
tions  research  effort  is  concerned  with  assisting  management  in 
decision  making.  This  statement  has  some  perhaps  not-so-obvious 
implications,  for  instances 

o  Analysts  should  seek  to  identify,  and  even  propose, 
feasible  alternatives. 

o  Analysts  should  discuss,  evaluate,  and  recommend  one  or 
more  alternatives. 

o  Analysts  should  clearly  delineate  the  points  at  which 
assumption  variation  modifies  findings  and  recommendations 
and  make  those  modifications  explicit. 

o  Analysts  should  include  implementation  methods  in  con¬ 
sidering  alternatives,  so  that  they  feasibily  accommodate 
environmental  realities. 

triangular  distribution.  This  procedure  of  using  standard 

multipliers  substantially  weakens  the  analysis,  and  its 

implications  should  be  discussed. 
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o  Analysts  should  remember  that  a  manager  wants  more  than 
a  digested  set  of  facts;  therefore  it  will  be  necessary 
to  make  assertive  proposals. 

6.  GENERAL 

The  documentation  guidelines  discussed  in  this  chapter  apply 
to  an  extensive  risk  assessment.  The  limited  application  of  risk 
assessment  for  budgeting  requires  somewhat  modified  documentation 
standards.  The  same  observation  can  be  made  about  applicability 
of  some  of  the  more  elaborate  risk  assessment  techniques,  yet 
both  the  elaborate  techniques  and  the  extensive  report  may  give 
rise  to  budgeting  decisions. 

Higher  review  levels  in  the  services  and  DOD  will  become 
increasingly  interested  in  evaluating  the  basis  of  budgets  incor¬ 
porating  risk  assessment  concepts.  Deputy  Secretary  Carlucci 
encouraged  the  Services  to  incorporate  technical  risk  considers- 
tion  into  budgeting,  and  the  Army  has  been  doing  so  for  some 
years.  Congress  is  aware  of  TRACE  and  of  DOD's  interest  in  the 
other  miliary  services’  adopting  similar  techniques;  thus  it  is 
likely  that  the  techniques  will  come  under  increasing  use  and 
scrutiny.  The  documentation  will  provide  confidence  in  the  basis 
of  budgets'  incorporation  of  risk  concepts.  Objective  evaluation 
and  acceptance  is  then  more  likely.  To  ensure  support  for 
budgets  an  abbreviated  documentation  should  cover  each  major  sub¬ 
ject  area  discussed  above  and  must  address  the  analytic  method  by 
which  findings  are  reached. 
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1 .  GENERAL 


POLICY  DIRECTIVES 


This  Appendix  summarizes  and  annotates  the  directives  of 
higher  authorities  and  the  military  Services. 

These  directives  were  identified  by  reviewing  the  system 
acquistion  and  management  directives  of  OMB,  DOD,  the  Army,  the 
Air  Force,  and  the  Navy,  and  noting  all  references  which  deal  spe¬ 
cifically  with  risk  analysis.  Some  references  imply  the  need  for 
risk  analysis,  but  do  not  explicitly  state  such  requirement. 
Although  additional  documents  were  reviewed,  only  those  listed 
were  found  to  contain  material  relevant  to  risk  analysis. 

2.  HIGHER  LEVEL  REQUIREMENTS 

This  section  lists  excerpts,  and  comments  briefly  on  OMB  and 
DOD  policies  and  directives  relating  to  risk  assessment. 

a.  Office  of  Management  and  Budget 

OMB  Circular  A-109.  Major  System  Acquisition  (5  April  1976) 

Para  7.  "Each  agency  acquiring  major  systems  should... 
tailor  an  acquisition  strategy  for  each  program. . . .The 
strategy  should  typically  include. . .methods  for  analyzing 
and  evaluating  contractor  and  Government  risks." 

b.  Department  of  Defense 

DOD  Directive  (DODD)  5000.1.  Major  System  Acquisition  (29 
March  1982) 

Para  C.2.C. (3) .  To  achieve  program  stability,  DOD  com¬ 
ponents  will  "estimate  and  budget  realistically,  and  fund 
adequately,  procurement  (research,  development,  and 
production),  logistics  and  manpower  for  major  systems. 
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Para  E. 4.C. ( 1) (a) .  This  paragraph  states  that  it  may  be 
reasonable  to  delay  Milestone  II  decision  until  some  develop¬ 
ment  efforts  are  accomplished  in  order  to  "reduce  risk  and 
uncertainty  before  the  committment  to  a  major  increase  in  the 
application  of  resources  toward  full-scale  development  is 
made . " 


Para  E.8.  "Commensurate  with  risk,  such  approaches  (to 
reduce  acquisition  time)  as  developing  separate  alternatives 
in  high-risk  areas;...  should  be  encouraged." 

DODI  5000.2  Major  Systems  Acquisition  Procedures  (March  8, 
1983) . 

No  references  to  risk  analysis  appear  in  body  of  text, 
however.  Paragraphs  D.3.e.(l)(a)  and  D.3.e.(2)(a)  refer  to  the 
need  for  System  Concept  Papers  (SCP's)  and  Decision  Coordinating 
Papers  (DCP's)  to  establish  and  identify  goals,  thresholds,  and 
threshold  ranges  [emphasis  supplied],  thus  recognizing  the  con¬ 
cept  of  risk. 

Enclosure  (4)  Format  for  SCP  and  DCP. 

"VIII.  Technological  Risks  of  Selected  Alternative. 
For  Milestone  I  (SCP),  identify  key  areas  of  technological 
risk  which  must  be  reduced  by  R&D  and  validated  by  T&E  before 
Milestone  II.  For  Milestone  II  (DCP),  discuss  T&E  results 
that  show  all  significant  risk  areas  have  been  resolved. 
Also  for  Milestone  II,  verify  that  technology  is  in  hand  and 
also  engineering  (rather  than  experimental)  effort  remains." 

DODI  5000.38.  Production  Readiness  Reviews  (24  January  1979) 

Para  A. 2.  "The  objective  of  a  PRR  [Program  Readiness 
Review]  is  to  verify  that  the  production,  design,  planning, 
and  associated  preparations  for  a  system  have  progressed  to 
the  point  where  a  production  commitment  can  be  made  without 
incurring  unacceptable  risks  of  breaching  thresholds  of  sche¬ 
dule,  performance,  cost,  or  other  established  criteria." 
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Para  E.4.  "The  DPESO  (DOD  Product  Engineering  Services 
Office)  independent  production  readiness  assessment  will  con¬ 
sist  of  objective  conclusions  based  on  the  findings  of  the 
PRR  and  other  investigations.  This  assessment  will  identify 
potential  problem  areas  which  constitute  production,  cost,  or 
schedule  risks.  Each  risk  will  be  expressed  in  terms  of  its 
relative  magnitude  and  potential  consequences."  [emphasis 
supplied] 

DODI  7041.3.  Economic  Analyses  and  Program  Evaluation  for 
Resource  Management  (October  19,  1972) 

Enclosure  (2) 

Para  B . 7 .  "Risk/Uncertainty  Analysis.  Risk  assessments 
will  be  made  to  determine  the  expectation  or  probability  that 
program/project  objectives  will  be  realized  by  following  a 
specific  course  of  action  with  constraints  of  time,  cost,  and 
technical  performance.  [emphasis  supplied]  Actual  costs  and 
outputs  of  many  DOD  projects  differ  from  those  expected  at 
the  time  of  decision.  For  those  cases,  and  in  particular  for 
major  weapon  systems  covered  by  a  Selected  Acquisition  Review 
Report  or  subject  to  review  by  the  Defense  System  Acquisition 
Review  Committee  (DSARC),  the  impact  which  could  result  from 
this  variability  should  be  evaluated." 

Para  B.7.a.  "Independent  parametric  cost  estimates  can 
provide  an  early  test  of  the  reasonableness  of  cost  esti¬ 
mates.  Independent  parametric  cost  estimates  will  be  made  at 
key  decision  points  for  major  weapon  systems,  e.g.,  during 
concept  formulation  and  prior  to  making  major  commitments  of 
funds  for  development  and  production.  These  estimates 
generally  consider  cost  at  high  levels  of  aggregation  and  are 
predicated  on  actual  historical  costs  encountered  in  like  or 
similar  programs.  As  such,  they  incorporate  costs  for 
expected  uncertainties  on  the  average.  (1)  Costs  should  be 
derived  by  parametric  techniques  and  expressed  as  feasible 
ranges  in  terms  of  the  parameters  which  drive  them.  It  is 


H-3 


cT 


most  important  that  estimates  be  presented  as  cost  ranges 
related  to  the  probable  values  of  system  parameters,  charac¬ 
teristics,  or  attributes  which  are  determined  by  costs, 
[emphasis  supplied]  (2)  These  estimates  will  be  available 
for  each  DSARC  review.  Parametric  estimates  will  be  derived 
independent  of  functional,  program  manager  or  contractor 
influence.  (3)  When  the  independent  parametric  cost  estimate 
differs  from  the  program  manager's  current  estimate,  the 
latter  estimate  will  be  used  for  economic  analysis/program 
evaluations.  Once  a  program  estimate  is  established  as  a 
baseline,  a  program/project  manager  will  manage  his  program 
within  that  limitation.  (4)  The  program  manager's  current 
estimate  will  be  an  assessment  of  the  ultimate  cost  expected 
for  a  program/project  including  undef initized  contingencies, 
[emphasis  supplied]  As  such,  the  program  manager's  current 
estimate  should  be  relatively  stable  over  long  periods  of 
time  and  not  change  with  small  incremental  changes  to  the 
approved  program,  funding  changes,  or  financial  fluctuations. 
To  the  extent  possible,  schedules  and  funding  should  be 
structured  to  accommodate  program  uncertainties  and  unfore¬ 
seen  problems.1'  [emphasis  supplied] 

Para  B.7.b.  "Special  degrees  of  risk/uncertainty  asso¬ 
ciated  with  a  particular  program/project,  may  be  pointed  out 
quantitatively  in  an  analysis  and  used  for  program  review 
purposes.  Probability  estimates  can  be  developed  by  testing 
the  sensitivity  of  key  variables  on  estimated  costs  and  per¬ 
formance.  The  probability  that  each  of  the  possible  cost  or 
output  estimates  may  be  realized  should  be  discussed  narra¬ 
tively  when  there  is  no  basis  for  a  quantitative  estimate. " 
[emphasis  supplied] 


Para  B.7.c.  Estimates  will  be  expressed  in  terms  of 
performance  thresholds,  goals,  or  ranges.  Program/project 
estimates  will  include  the  limits  within  which  ultimate 
program  cost  and  technical  performance  is  expected  to  fall." 


3.  SERVICE  REQUIREMENTS 
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a.  U.S.  ARMY 

The  following  Army  directives  require  or  imply  a  need  for 
risk  assessment  as  shown  by  excerpts  and  editorial  condensations. 

o  Army  Regulation  70-1  Army  Material  Acquisition  (Draft,  14 
May  1981)  . 

Para  4-6. a. 1.  Advanced,  engineering  and  operational 
systems  development  is  to  "conduct  system  advanced  develop¬ 
ment  in  promising  areas  in  order  to  resolve  key  technical, 
readiness,  cost  and/or  schedule  uncertainties  before  entering 
the  full-scale  Development  Phase.  Such  efforts  should  be 
accomplished  with  low-level  programs  and  full  realization  of 
technical  risks." 

Para  4-10. a.  "In  general,  PEP  [ Producibility  Engineer¬ 
ing  and  Planning]  measures  include...  performing  risk  analy¬ 
sis  of  new  manufacturing  processes...  When  a  high-risk  pro¬ 
ducibility  area  is  determined  during  PEP  which  normal  design 
tradeoff...  cannot  provide  for  the  same  performance...,  and  a 
requirement  is  established  for  special-purpose  or  unique 
tooling  and/or  processes,  a  Manufacturing  Technology  Develop¬ 
ment  effort  ( RDTE  funded)  should  be  initiated.  If  a  high- 
risk  area  manufacturing  technique  that  is  generic  in 
nature...  is  identified,  a  manufacturing  Methods  and  Tech¬ 
nology  effort  ( MM&T  procurement-funded)  should  be  consid¬ 
ered." 

Para  5-2. c. (2) .  The  Management  Plan  for  System 
Development  will  be  supported  by  management  plans  for  risk 
analysis  among  other  plans. 

Para  5-2. c. ( 3) .  The  Financial  Plan  for  System  Develop¬ 
ment  will  include  "established  program  cost  controls  or  cost 
parameters  including  design-to-cost  goals  and  TRACE." 
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Para  8-5. f.  "Risk  analysis  and  management,  which  in¬ 
cludes  elements  of  risk  identification,  planning,  analysis, 
evaluation,  resolution,  and  review,  will  be  completed  for 
computer  resources  prior  to  Milestone  II.  Computer  resource 
risk  analysis  and  management  will  be  closely  coordinated  with 
the  requirements  validation  effort  and  the  overall  systems 
engineering  effort  to  assure  that  the  risks  associated  with 
achieving  validated  cost,  schedule,  and  technical  performance 
requirements  are  identified  and  assessed  in  advance,  are 
within  acceptable  thresholds,  and  are  continuously  monitored 
and  reported  during  subsequent  development." 

o  Army  Regulation  (AR)  70-10.  Research  and  Development,  Test 
and  Evaluation  During  Development  and  Acquisition  of  Materiel 
(29  August  1975). 

Para  1-4  .  ( 3 )  .  "During  the  full-scale  development  phase 
and  prior  to  the  first  major  production  decision,  the  DT 
(Development  Test)  accomplished  will  be  sufficiently  adequate 
to  insure  ...  that  all  significant  design  problems  (including 
compatibility,  interoperability,  safety.  Reliability,  Availa¬ 
bility,  and  Maintainability  (RAM),  and  supportabili ty  consid¬ 
erations)  have  been  identified,  and  that  solutions  to  the 
above  problems  are  in  hand. 

Para  2-4  ,c .  ( 5 ) .  "As  the  development  cycle  continues  into 
DT/OT  [Development  Test/Opertional  Test]  II,  the  uncertainty 
in  these  estimates  should  be  reduced  and,  by  the  completion 
of  DT/OT  III,  sufficient  testing  should  have  been  accom¬ 
plished  so  that  the  uncertainty  in  estimating  the  final  sys¬ 
tem  performance  will  be  relatively  small,  ...." 

Para  2-5. e.  "EDT  (Engineering  Development  Test)  is  con¬ 
ducted  by  the  contractor  and/or  the  material  developer  with 
the  primary  objective  of  influencing  material  design."  ... 
"The  purposes  of  EDT  are  to... (3)  Eliminate  as  many  technical 
and  design  risks  as  possible  or  to  determine  the  extent  to 
which  they  are  manageable." 


f. 
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o  Department  of  the  Army  Pamphlet  (DA-Pam)  11-2.  Research  and 
Development  Cost  Guide  for  Army  Materiel  Systems  (May  1976). 


Para  3.5.  Range  Versus  Point  Estimates. 


"The  use  of 


a  point  estimate  does  not  reflect  the  uncertainty  associated 
with  the  estimate.  It  also  implies  that  it  is  a  precise 
cost.  For  these  reasons,  a  range  of  costs  should  be  provided 
based  on  the  inherent  cost  estimating  uncertainty.  The  level 
at  which  the  ranges  can  be  provided  is  dependent  upon  the 
level  at  which  the  costs  are  estimated.  Within  the  limita¬ 
tions  imposed  by  the  data  base  and  cost  estimating  approach 
employed,  ranges  should  be  presented  at  the  highest  aggregate 
level . 

"b.  In  addition,  an  analysis  should  be  made  of  the  sen¬ 
sitivity  of  projected  costs  to  all  critical  assumptions. 
This  should  include  factors  such  as  the  impact  of  changes  in 
performance  characteristics,  changes  in  configuration  to  meet 
performance  requirements,  schedule  alternations  and  alter¬ 
native  production  processes." 

o  DA-Pam  11-3.  Investment  Cost  Guide  for  Army  Materiel 
Systems  (April  1976)  Exactly  the  same  words  as  above. 


o  DA-Pam  11-4.  Operating  and  Support  Cost  Guide  for  Army 
Materiel  Systems  (April  1976).  Exactly  the  same  words  as  above. 

o  Development  Acquisition  and  Readiness  Command  Regulation 
(DARCOM-R)  11-1.  Systems  Analysis 


Para  4.d.  "...RA  and  DRA  are  applied  to  alternative 
courses  of  action  and  permit  structuring  models  that  address 
the  uncertainty  of  cost,  schedule,  and  performance  of 
systems."  [RA  and  DRA  mean  risk  analysis  and  decision  risk 
analysis  respectively.] 

Para  5.c.  "An  IE  and  DRA  will  be  completed  prior  to 
each  decision  milestone  in  major  programs  which  will  involve 
....  (ASARC)  or  ....  ( DSARC )  proceedings,  or  in  non-major 


S 


i 
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programs  for  which  DA  has  retained  in-process  review  (IPR) 
approval  authority.  For  non-major  systems,  an  ASARC  will  be 
completed  prior  to  each  IPR  "unless  it  is  clear  that  no 
appreciable  time,  cost,  or  performance  risk  is  associated 
with  the  decision."  [IE  means  Independent  Estimate;  ASARC, 
Army  Systems  Acquisition  Review  Council;  DSARC,  Defense 
Systems  Acquisition  Review  Council;  DA,  Department  of  the 
Army . ] 

Para  6.d.  "Each  commander  of  an  R&D  or  MR  Command  wills 
3)  ensure  that  IE's  and  DRA's  are  initiated  for  the  decision 
points  described  in  paragraph  5c."  [MR  means  Materiel  Readi¬ 
ness  .  ] 

Appendix  D-Decision  Risk  Analysis  Guidelines 

1.  Define  the  problem. 

b.  Establish  the  decision  maker's  preferences  for 
tradeoffs  between  cost,  schedule,  and/or  perfor¬ 
mance. 

2.  Establish  the  alternatives. 

3.  Define  the  events. 

4.  Collect  the  data. 

5.  Determine  the  program  risks. 

6.  Select  the  best  alternatives. 

7.  Perform  sensitivity  analysis. 

8.  Communicate  the  results. 

o  DARCOM-R  11-27.  Life  Cycle  Management  of  DARCOM  Materiel 
Chapter  3 .  Section  II  -  Procedures 

Para  3-8. b.  "The  justifications  for  by-passing  activi¬ 
ties  and  events  are...  (2)  That  the  risk  for  omitting  the 
actions  is  reasonable  when  considering  the  savings  of  time 
and  resources." 

Para  3-8.c.  "When  requesting  the  shortening  of  sched¬ 
ules,  the  PM  will  request  DARCOM  approval  and  submit  a  state¬ 
ment  including  the  assessment  of  risks  incurred  in  shortened 

plan  compared  to  base  plan." 
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o  AR-11-28 .  Economic  Analysis  and  Program  Evaluation  for 
Resource  Management 


Chapter  1 

Para  1-6. p.  "Where  costs  for  research  and  development 
represent  a  significant  portion  of  total  program  cost,  the 
decision  to  conduct  research  will  be  supported  by  an  economic 
analysis  which  identifies  potential  follow-on  cost,  savings 
resulting  from  the  research  and  development,  degree  of  risk 
or  uncertainty  in  achieving  results,  availability  of  resour¬ 
ces,  assessment  of  current  technology,  and  identification  of 
constraints . 


Chapter  2 

Para  2-2. b.  "The  structure  of  analysis  will  also  con¬ 
tain,  when  appropriate,  an  assessment  of  the  relative  risk 
or  uncertainty  of  success  associated  with  each  of  the  alter¬ 
natives  considered,  including  the  status  quo  when  applicable." 


o  AR-15-14.  Boards,  Commissions  and  Committees 
Accruisition  Review  Council  Procedures 


Systems 


Page  4-3.  This  paragraph  states  that  risk  analysis  will 
be  presented  to  HQ DA  2  months  before  ASARC. 

o  AR-70-1.  Army  Research,  Development  and  Acquisition 
(29  August  1975) 

Para  1-7. o.  "Technical  uncertainty  will  be  continuously 
assessed.  Progressive  commitments  of  resources  will  be  made 
only  when  confidence  in  program  outcome  is  sufficiently  high 
to  warrant  going  ahead." 

Para  2-2. a.  In  conceptual  phase,  "critical  technical 
issues,  operational  issues,  and  logistical  support  problems 
are  identified  for  resolution  in  subsequent  phases  in  order 
to  minimize  future  development  risks." 


Para  2-15. a.  "Test  and  evaluation  will  be  conducted  as 
early  as  possible  and  throughout  the  material  acquisition 
process  to  reduce  acquisition  risks...." 

Para  2-15. c(4).  DT  will  be  used  to  "demonstrate,  during 
the  Full  Scale  Development  Phase  and  prior  to  the  first  major 
production  decision,  that  the  DT  accomplished  is  adequate  to 
insure  that  engineering  is  reasonably  complete;  that  all 
significant  design  problems. . .have  been  identified;  and  that 
solutions  to  the  above  problems  are  in  hand." 

Para  4-la.  "Department  of  the  Army  policy  for  advanced, 
engineering,  and  operational  systems  development  is  to — (1) 
Conduct  system  advanced  development  in  promising  areas  using 
either  single  or  competitive  approaches  in  order  to  resolve 
key  technical,  cost  and/or  schedule  uncertainties  before 
entering  Full-Scale  Development  Phase.  Such  efforts  should 
be  accomplished  with  low-level  programs  and  full  realization 
of  technical  risks." 

Para  4-lm.  "Program  sufficient  funds  to  provide  for  the 
technical  uncertainty  inherent  in  the  development  effort." 

Para  4-laf.  "Give  consideration  to  requiring  develop¬ 
ment  contractors  to  provide  second  sources  for  high  technical 
risk  subsystems/components,  whether  or  not  the  development 
contract  is  sole  source  or  competitive." 

o  AR  71-9.  Material  Objectives  and  Requirements  (Final 
Draft  26  Feb  1981). 

Para  1-5  "Test  and  Evaluation  will  begin  as  early  as 
possible  in  the  acquisition  cycle  and  will  be  conducted 
throughout  the  system  acquisition  process  as  necessary  to 
assess  acquisition  risks  ..." 


Para  3-4. k.  "Army  R&D  organizations  are  to  take  the 
following  specific  actions  with  respect  to  the  STOG  (Science 
and  Technology  Objectives  Guide):  (1)  Perform  assessments 


with  respect  to  overall  feasibility,  estimated  cost,  time, 
risks,  and  possible  technological  solution  alternatives 
within  mission  areas....  (4)  Communicate  directly  with  the 
user  proponents  on  topics  such  as  ....(c)  Informing  the 
user  proponent  of  estimated  solution  alternatives,  risks, 
time  and  estimated  manpower  and  dollar  costs,  so  that  the 
user  can  perform  an  affordability  and/or  tradeoff  assess¬ 
ment  . " 


Para  4-2. d. (7) (b) .  "...Even  when  pursuing  a  single  con¬ 
cept,  competition  should  be  considered  in  development  of  the 
concept  or  component  development  in  order  to  minimize  risk  of 
hardware  development . " 


o  Letter  of  Instruction  (LOI)  for  Implementation  of  RD&TE 
Cost  Realism  for  Current  and  Future  Development  Programs, 
DAMA-PPM-P,  (6  March  1975). 


Para  3. a.  Defines  TRACE  as  "the  expected  total  cost 
over  a  specified  period  of  a  material  development  program 
computed  on  the  basis  of  the  costs  of  accomplishing  the  work 
elements  of  the  program's  Work  Breakdown  Structure...,  and 
including  specific  provisions  for  the  statistical  estimation 
of  probable  program  costs  otherwise  indeterminate.  The  TRACE 
should  be  that  estimate  having  a  50/50  chance  of  producing 
either  a  cost  overrun  or  an  underrun." 


Para  3.d.  Defines  risk  factor  as  "as  assessment  of  the 
probable  expansion  of  that  work  necessitated  by  changes  in 
other  elements  of  the  program  or  by  changes  in  the  work 
itself." 


Para  3.e.  Defines  risk  analysis  as  "an  analysis  con¬ 
ducted  as  a  part  of  the  cost  estimation  process  for  a 
material  development  program  in  which  risk  factors  are 
constructed  encompassing  probable  cost  increases  to  a  program 
work  element  expected  during  its  development,  ..." 


Para  4,b.  "Specifically,  the  TRACE  should  include  con¬ 
sideration  of  the  following: 

(1)  Technical  design  changes  to  correct  deficiencies. 

(2)  Design  changes  to  accommodate  nominal  revisions  in 
component  performance. 

(3)  Rescheduling  to  work  around  technical  problems  and 
nominal  budetary  limitation. 

(4)  Additional  testing  to  verify  design  corrections. 

(5)  Additional  hardware  to  support  design  modifica¬ 


tions  . 


(6)  Non-neglegent  human  error." 


Para  4.c.  "The  TRACE  will  be  the  material  development 
program  cost  estimate  used  for  program  planning  and 
justification. " 


Para  5. a. (1) .  "It  is  to  be  emphasized  that  the  purpose 
of  the  TRACE  is  not  to  create  unjustified  'reserves.'  Rather, 
the  TRACE  is  to  produce  realistic  cost  estimates  of  what  pro¬ 
bably  will  be  required  on  a  development  program. " 


Para  5.a.(2).  "In  the  conduct  of  risk  analysis,  each 
Work  Breakdown  Structure  (WBS)  element  identified  in  a 
materiel  development  program  will  be  analyzed  to  determine 
the  specific  sub-elements  that  will  contribute  to  uncertainty 
in  the  establishment  of  the  cost  of  execution  of  that  ele¬ 


ment.  A  risk  factor  will  then  be  constructed  for  each  iden¬ 


tified  WBS  element  as  a  number  increased  over  1  by  an  amount 
representing  an  accumulation  of  estimated  proportional 
increases  in  the  cost  of  that  element  as  a  result  of  probable 
changes  to  it  during  its  development.  These  increases  will 
be  adjusted  by  a  judgmental  determination  of  just  how  likely 
it  is  that  each  will  occur." 


Para  5. a. (3) ♦  "If  the  specific  program  work  has  not  yet 
been  identified  in  sufficient  detail,  risk  factors  will  be 
assigned  on  the  basis  of  larger  aggregates  of  work,  system 
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components,  or  system  types,  depending  upon  the  detail 
available.  In  such  cases,  risk  factors  will  be  constructed 
judgmentally  in  full  consideration  of  the  engineering,  produ- 
cibility  and  budgetary  aspects  of  the  program.  Specific  con¬ 
siderations  to  be  included  in  this  judgment  are: 

(a)  Whether  the  program  requires  the  development  of  an 
item  not  directly  supported  as  feasible  by  existing 
technology. 

(b)  Whether  the  program  requires  the  development  of  an 
item  substantially  different  from  those  previously 
developed. 

(c)  Whether  major  integration  effort  will  be  necessary 
even  though  individual  components  may  in  themselves 
be  considered  to  involve  low  risk." 

Para  5.b.  "TRACE  computation.  The  risk  factors  will  be 
multiplied  by  the  engineering  cost  estimate  at  the  appro¬ 
priate  level  of  the  WBS.  The  appropriate  level  will  depend 
not  only  on  the  level  of  design  detail  available,  but  also  on 
the  degree  of  component  and  subsystem  interaction.  In  those 
circumstances  where  a  design  change  of  a  given  component  or 
subsystem  appears  likely  to  propagate  and  cause  a  design 
change  of  a  related  component  or  subsystem,  a  higher  level  of 
aggregation  will  also  be  required  to  maintain  statistical 
validity  of  the  overall  estimate  by  including  these  inter¬ 
dependent  effects.  The  risk  factors,  when  applied  at  the 
appropriate  level  of  the  WBS  as  explained  above,  can  be  sta¬ 
tistically  combined  to  produce  the  TRACE." 

Para  6. a.  "The  costs  of  specific  program  work  scheduled 
for  accomplishment  during  a  particular  year  will  be  estimated 
using  the  TRACE  methodology.  The  TRACE  thus  compiled  for  the 
program  'work-year'  will  be  the  amount  submitted  to  OSD  and 
the  Congress  as  required  for  the  program  for  that  year.  The 
funds  representing  the  difference  between  the  TRACE  and  the 
engineering  cost  estimate  will  not  be  carried  in  a  separate 
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category,  but  rather  will  be  allocated  to  the  various  tasks 
to  which  the  funds  will  most  likely  be  applied." 

Para  6,b.  "To  allow  for  the  possibility  of  cost  sav¬ 
ings  to  allow  more  precise  managerial  control  of  funds 
appropriated  for  program  execution  during  a  budget  year,  only 
that  amount  reflecting  the  basic  (engineering)  cost  estimated 
for  that  year  (i.e.,  the  engineering  cost  estimate  consisting 
of  the  work  costs  prior  to  multiplication  by  the  respective 
risk  factors)  will  be  released  initially  to  the  manager  for 
program  execution.  The  remainder  of  the  appropriated  program 
funds  will  be  held  in  deferral  by  the  DCSRDA  and  released  to 
the  manager  only  upon  request  and  approval  of  a  justified 
need.  Program  funds  (obligated  authority)  not  required  in 
the  current  year  program  will  be  considered  for  designation 
to  the  Congress  as  a  means  to  reduce  the  requirement  for  new 
obligational  authority  is  the  subsequent  year's  budget. 
Other  use  of  such  unneeded  funds  may  be  authorized  by  the 
DCSRDA,  as  appropriate." 

o  Letter  of  Instruction  (LOI)  for  Implementation  of  the 
Total  Risk  Assessing  Cost  Estimate  for  Production  (TRACE-P)  (6 
October  1982). 

Para  4.b.  "The  TRACE-P  estimate  will  include  considera¬ 
tion  of  the  risks  in  the  following  categories;  these  are 
explained  at  Enclosure  1. 

(1)  Threat  Uncertainty 

( 2 )  Management 

(3)  Materials/Purchased  Parts 

(4)  Facilities/Equipment 

(5)  Labor 

(6)  Design  Changes 

(7)  Producibility 

(8)  Performance" 

Para  4.c.  "Specifically  excluded  from  the  estimating  of 
TRACE-P  expected  risk  costs  are  possible  increases  that  may 
result  from  one  or  more  of  the  following  causes: 
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(1)  Quantity  changes 

(2)  Performance  improvement  to  meet  an  increased  threat 

( 3 )  Poor  management 

(4)  Inadequate  funding  in  the  early  years 

(5)  Unknown  unknowns" 

Para  4.g.  "TRACE-P  funds  will  be  held  in  deferral  by 
the  DCSRDA  and  released  to  the  program/project/product 
manager  only  upon  request  and  approval  of  a  justified  need." 

o  RDA  Cost  Realism  -  Future  Development  Programs  (12  July 
1974)  DASA  Letter 

"Our  estimate  should  be  unbiased  so  that  we  have  about  an 
even  chance  of  either  going  over  or  under  it." 

"It  is  submitted  that  cost  overruns  will  continue  to  be  a  way 
of  life  until  adequate  recognition  is  given  to  the  impact  of 
program  uncertainty  in  estimating  costs"  [emphasis  in  origi¬ 
nal]  . 

"...  it  remains  the  fundamental  nature  of  RDT&E  ...  to  involve 
the  unknown.  These  unknowns  invariably  lead  to  cost  require¬ 
ments  which  cannot  be  individually  forseen  at  the  outset  of  a 
development — yet  their  cumulative  impact  can  be  seen  in 
retrospect  with  all  the  assuredness  of  the  laws  of  probabil¬ 
ity"  [emphasis  in  original]. 

"The  provision  of  flexibility  in  the  funding  plan  baseline- 
cost  estimates  should  reflect  these  probable  additional 
costs . " 

o  Total  Risk  Assessing  Cost  Estimate  (TRACE)  Deferrals 
DRCDEPC  letter,  (17  April  1978). 

Para  2.  "A  program  TRACE  refers  to  its  total  expected 
RDT&E  costs  as  agreed  to  by  the  ASARC/DSARC.  The  definition 
applies  both  to  annual  costs  and  development  costs.  Funds  in 
excess  of  the  baseline  or  engineering  costs  for  a  particular 
fiscal  year  are  deferred  at  the  beginning  of  that  year  by 
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HQDA  (DCSRDA)  pending  the  occurrence  of  predicted  (but  unpro- 
g rammed)  events  upon  which  the  funds  were  based.  These  funds 
will  be  released  to  the  Project  Manager  (PM)  upon  his  demon¬ 
stration  that  they  are  necessary  to  offset  the  cost  of  such 
events.  If  a  program  adjustment  is  made  during  the  program¬ 
ming  and  budgeting  cycle,  the  entire  scope  of  the  program 
should  be  reevaluated  and  the  risk  factor  recomputed.  TRACE 
funds  identified  for  deferral  in  the  outyears  should  not  be 
reduced  in  order  to  increase  the  baseline  portion  if  that 
adjustment  is  made  merely  to  offset  a  decrement  to  the 
program  or  to  increase  its  scope." 

Para  3 .  "The  funds  released  from  HQDA  ere  expected  to 
be  adequate  for  execution  of  *the  known  estimated  engineering 
costs.  It  is  DARCOM  policy  to  expect  that  the  PM  manage  his 
total  program  with  the  funds  authorized.  Risk  contingency 
(TRACE  deferral)  funds  will  be  released  only  if  technical/ 
design  problems  and/or  unexpected  delays  materialize,  and 
this  fact  is  completely  demonstrated  in  the  release  request. 
Release  requires  both  DCSRDA  and  ASA  (R&D)  approval." 

Para  4.  "The  PM  can  indicate  at  any  time  that  TRACE 
deferral  funds  will  not  be  needed.  If  deferral  has  not  been 
released  by  the  fifth  quarter  of  availability,  the  PM  will  be 
given  the  opportunity  to  justify  retention  of  such  deferrals 
considering  that  the  work  upon  which  they  were  based  may  have 
continued  into  the  second  year.  Otherwise,  disposition  of 
the  funds  will  be  determined  by  the  DCSRDA  in  coordination 
with  the  ASA  (RL&LD)  and  CDR  DARCOM.  The  PM  may  request 
that  the  unneeded  TRACE  deferral  funds  be  released  for 
expanded  scope  of  work  in  the  same  weapons  systems;  however, 
that  request  will  be  evaluated  with  other  unfinanced  require¬ 
ments  in  other  programs." 

b.  US  Air  Force 


The  following  Air  Force  directives  address  consideration  of 
program  risk  as  revealed  by  excerpt  or  editorial  summarization. 
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sis  Program  (12  Dec  1980) 


Para  6 .  Definition  and  Scope  of  the  Independent  Cost 
Analysis  (ICA) 

Para  6.f.  "Will  contain  a  detailed  risk  assessment  to 
include  risk  related  to  the  cost  estimating  techniques  em¬ 
ployed  and  with  technical  and  schedule  uncertainties  that  may 
have  an  impact  on  cost  estimates.  It  will  also  include  sen¬ 
sitivity  analyses  of  critical  assumptions  and  cost  during 
parameters. " 

Para  7.e.  "For  cost  elements  with  a  high  degree  of 
uncertainty,  the  ICA  will  provide  sensitivity  analysis  using 
frequency  distributions  or  ranges  of  cost.  The  probability 
distributions  used  to  prepare  range  estimates,  as  well  as  the 
proper  assumptions,  must  be  provided.  Prediction  intervals 
around  cost  estimating  relationships  (CERs)  or  Monte  Carlo 
simulations  will  be  used  as  proper  in  quantifying  risk. " 
[emphasis  supplied] 

Para  9.d.  "The  ISR  will  address  the  potential  risk  in 
the  program  office  estimate  by  identifying  'risk'  areas  and 
their  probable  and  possible  cost  impact."  [ISR  means 
Independent  Schedule  Review] 

o  AFR  70-15.  Source  Selection  Policy  and  Procedures 

Para  1-4. d.  "The  source  selection  process  shall  focus 
adequate  attention  on  the  program  risk  and  uncertainties 
during  solicitation,  proposed  evaluation,  and  selection  pha¬ 
ses  . 


(1)  Offerors  should  not  be  penalized  for  the  iden¬ 
tification  of  risk  associated  with  their  proposals. 
Proposals  should  be  credited  when  realistic 
approaches  for  risk  resolution  are  provided. 


(2)  The  procuring  activity  shall  prepare  an  independent 
risk  assessment  before  receipt  of  proposals,  to 
facilitate  risk  analysis  evaluation. 

Para  2-2. c(3).  "It  [the  evaluation  criteria]  must 
address  those  high  risks  and  technical  uncertainties,  which 
were  identified  by  the  offerors  and  the  Government  as 
'known-unknowns'  during  the  conceptual  phase.  An  indication 
should  also  be  provided  of  the  relative  importance  of  each 
criterion  for  later  use  in  the  solicitation. 


Ha 


Para  2-4. d.  "...  Risk  analysis  is  a  part  of  the  eva¬ 
luation  process,  and  risk  assessment  for  each  proposal  must 
be  included  in  all  reports  to  the  SSAC  [Source  Selection 
Advisory  Council]  and  SSA  [Source  Selection  Authority]. 
Technical  risk,  as  pertains  to  each  proposal,  should  be  rated 
based  on  the  offeror's  risk  assessment  and  the  credibility  of 
his  proposed  approach  for  eliminating  or  avoiding  such 
risks. " 

Para  3-2.a.(2).  "The  solicitation  should  . . .  include  a 
discussion  of  known  or  potential  risks,  where  there  is  rea¬ 
son  to  believe  that  the  potential  offerors  are  not  aware  of 
the  risks." 

Para  3-7. e.  "The  offerors  must  be  required  to  submit  a 
risk  analysis  as  part  of  their  proposal  which  also  identifies 
risk  areas  and  which  furnishes  an  insight  to  the  evaluator  as 
to  how  the  offeror  intends  to  resolve  these  risks  and  the 
alternatives  to  overcoming  the  high  risk  approaches.  (1)  In 
order  to  aid  the  evaluator  in  performing  the  risk  analysis, 
the  procuring  activity  should  prepare  an  independent  risk 
assessment  prior  to  receipt  of  proposals." 

Para  3-8 ,b. (5) .  This  paragraph  states  that  the  SSA  must 
determine  cost/price  risk  inherent  in  each  proposal. 
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Attachment  4 
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'  VIII.  Risk  Analysis 

This  paragraph  lists  risk  analysis  documentation  format. 

o  AFP  800-3.  Engineering  for  Defense  Systems,  (17  June 
1977) . 

Para  4.b.  [In  the  validation  phase] ".. .certain  technical 
aspects  may  need  to  be  intensified,  such  as  technical  and 
cost  risk  reduction,  obtaining  a  best  mix  of  technical 
requirements,  and  other  considerations  or  thresholds  as  may 
be  described  in  the  PMD." 

Para  6.f.  The  AFSC  "programs  their  research  and  devel¬ 
opment  (R&D)  projects  to  develop  and  improve  systems  engi¬ 
neering  methods  and  techniques  (system  cost  effectiveness, 
risk  assessment,  technical  performance  measurement,  etc.)." 

o  AFR  800-8.  ILS  Program,  (7  February  1980). 

Para  5 .r .  "Risk  analysis  and  assessment  and  tradeoff 
analyses  will  be  conducted,  using  the  latest  data  available." 

o  AFR  800-9.  Manufacturing  Management  for  Air  Force  Ac¬ 
quisitions  (1  October  1979). 

Para  2.c.  "In  the  manufacturing  assessment  of  system 
and  design  alternatives  the  program  manager  will:  (1) 

Consider  the  relationship  between  several  factors  (such  as 
producibility,  manufacturing  risks,  productivity  ....)  and 
evaluate  their  impact  on  the  minimum  essential  performance 
requirements . " 

Attachment  1.  Extracts  from  DODD  5000.34,  31  October  1977. 

Para  D.5.  "...  Production  risks,  which  should  be  iden¬ 

tified  as  early  as  possible  in  the  acquisition  cycle,  shall 
be  reduced  to  acceptable  levels  prior  to  production 
decision. " 
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o  Aeronautical  Systems  Division  Regulation  (ASDR)  173-1 
Aeronautical  Systems  Division  Cost  Analysis  Program  (21  October 
1981) . 

Attachment  5:  Cost  Estimate  Risk  Assessment  Guidelines 

Para  1.  "...The  purpose  of  the  risk  analysis  described 

below  is  to  alert  decision  makers  to: 

a.  Inputs  or  assumptions  where  a  percent  change  in  an 
input  or  assumption  value  would  make  at  least  one 
half  that  percent  change  in  the  total  estimate. 

b.  Areas  of  uncertainty  at  the  time  the  estimate  was 
prepared. . . . 

Para  2 .  "Generally,  risk  assessments  must  be  prepared 
so  that  while  not  all  possible  areas  of  risk  are  addressed, 
the  overall  amount  of  risk  of  cost  growth  can  be  assessed  by 
review  of  the  several  highest  risk  areas  identified  and 
discussed. " 

Para  3.  "The  risks  of  cost  increase  over  the  estimates 
to  be  addressed  will  be  primarily  those  associated  with  esti¬ 
mating  methods  are  [sic]  available  data/information  limitations. 
The  risks  of  strikes,  major  test  or  technical  approach  fail¬ 
ures,  directed  program  changes,  etc.,  are  not  to  be  ad¬ 
dressed.  " 

c.  US  Navy 

The  following  Navy  directives  address  program  evaluation  in¬ 
cluding  some  mention  of  risk  evaluation.  Excerpt  and  editorial 
summaries  are  presented. 

o  Secretary  of  the  Navy  Instruction  (SECNAVINST)  5000. IB 
System  Acquisition  (8  April  1983). 

Para  5.q.  Management  Principles  and  Objectives.  The 
instruction  presents  as  a  management  principle  "applying 
established  or  evolving  technology  having  a  high  probability 
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of  success.  High  technical  risks  may  be  taken  if  an  extraor¬ 
dinary  payoff  potential  can  be  demonstrated." 

Para  6.  "Acquisition  Categories.  A  program  is  a  candi¬ 
date:  ACAT1IS  designation  by  SECNAV,...  if  it  is  of  special 
SECNAV  interest,  ...because  of...  a  history  of  technical, 
cost  and  schedule  problems,"  or  "an  extraordinary  strategy 
and/or  risks." 

Para  8.  "Decision  Milestones.  Milestones  and  phases 
will  be  tailored  to  fit  each  program  to  reduce  acquisition 
time  and  cost,  consistent  with  risk." 


Para  8.b.  "Milestone  II, 


It  should  be  demonstrated  to 


the  decision  authority  that  technical  and  operational  risks 
have  been  reduced  to  acceptable  levels." 

Enclosure  (2)  Management  Considerations 

Para  3.  "Acquisition  Time.  Programs  shall  be  planned  for 
system  development  within  the  shortest  time  reasonable.  At 
each  milestone,  schedule  alternatives  and  inherent  risks 
shall  be  assessed.  Methods  to  be  considered  include  com¬ 
bination  or  omission  of  acquisition  phases;  smooth  transition 
to  production;  single  concept  development;  preplanned  product 
improvement;  use  of  alternatives  in  high  risk  areas;  experi¬ 
mental  prototyping  of  critical  components;  or  coordination  of 
common  purchases  between  different  programs." 

Para  10.  "Test  and  Evaluation.  Test  and  evaluation  are  an 
integral  part  of  the  acquisition  process  to  assess  technical 
performance  and  risks,...  Schedules  shall  be  flexible  to  allow 
retest  or  reevaluation  as  necessary  prior  to  a  milestone,  and 
shall  avoid  duplication  commensurate  with  risk." 

Para  14.  "Acquisition  Risks.  Technical,  operational,  sche¬ 
dule,  and  cost  risks  shall  be  identified  as  early  as  possible 
and  assessed  continuously.  They  shall  be  disclosed  in  full 

to  the  decision  authority  and  addressed  realistically  at  each 
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milestone.  A  management  reserve  bases  on  the  cost  risk  shall 
be  established  for  ACAT  I  and  IIS  programs." 


Enclosure  (4)  Navy  Decision  Coordinating  Paper  (NDCP)  Format 


Para  1.  "Risks .  State  program  risk,  including  at  Milestone 
I,  technological  risks  to  be  reduced  by  R&D  and  validated  by 
T&E  before  Milestone  II;  at  Milestone  II,  demonstrate  that 
all  significant  risk  areas  have  been  resolved  and  verify  that 
technology  is  in  hand  and  only  engineering  (rather  than 
experimental)  effort  remains;  at  Milestone  III,  identify  any 
shortfalls  in  technical  evaluation  (TECHEVAL)  and  OPEVAL 
results  against  thresholds. 


o  Naval  Material  Command  Instruction  (NAVMATINST)  5000. 19D 
Acquisition  Program  Reviews  and  Appraisal  Within  the  Naval 
Materiel  Command  (16  February  1982). 


Para  4a.  "...  this  instruction  requires  a  consistent, 
timely,  independent  assessment  of  major  programs  in  order  to 
ensure  that  they  are  technically,  financially,  administra¬ 
tively  and  logistically  sound.  The  (independent)  assessment 
is  to  provide  a  high  degree  of  confidence  that  the  program 
is,  in  fact,  sound  and  executable." 


Para  4.b.  "The  ARB's  [Acquisition  Review  Board's]  pur¬ 
pose  is  to  ensure  that  the  presentation  accurately  reflects 
the  Chief  of  Naval  Material  position,  that  the  program  itself 
is  logical  and  executable  from  a  business/programmatic  and 
technological  standpoint  and  complies  with  applicable  tasking 
from  higher  authority.  " 


o  Naval  Material  Command  Instruction  (NAVMATINST)  5000. 29A 
Acquisition  Strategy  Paper  (6  May  1983). 


Para  2.  The  Acquisition  Strategy  Paper  shall  discuss 
Risk  Analysis  in  Section  II  -  Risk  Analysis  Enclsoure  (1) 
"specify  the  major  problems  or  risk  areas  which  have  been 
considered  in  selection  of  an  acquisition  strategy  and  which 
must  be  overcome  to  achieve  the  basic  program  objectives." 
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Para,  3.  Section  III  -  Strategy  to  Achieve  Objectives 
and  implementation  shall  contain  the  "Risk  Management  Plan 
for  dealing  with  areas  (technical,  costs,  schedule  and 
logistics),"  and  the  business  management  plan  of  "incentives 
to  achieve  program  thresholds  including  methods  to  control 
costs,"  and  "incentives  to  improve  reliability  and  reduce 
support  costs." 

o  Naval  Air  Systems  Command  Instruction  (NAVAIRINST)  7131.1 
Management  of  Research,  Development,  Test  and  Evaluation,  Navy 
(RDT&E,N)  Risk  Cost  Estimate  Funding  (21  April  1983). 

Para  4.  "Procedures.  BCE  (baseline  cost  estimate)  funds 
budgeted  and  approved  for  a  project  will  be  allocated  to  the 
requiring  financial  manager  (RPM)  in  accordance  with  estab¬ 
lished  procedures.  It  is  expected  that  BCE  funds  will  be 
adequate  for  the  execution  of  known  requirements  and  that  the 
RFMs  will  manage  their  programs  within  funds  authorized.  RCE 
(risk  costs  estimate)  funds  will  be  held  by  the  RDT&E  Budget 
Division  (AIR-803)  and  released  only  it  technical,  design,  or 
other  unexpected  problems  arise,  and  the  facts  are  completely 
justified.  Release  of  RCE  funds  requires  the  approval  of 
both  the  cognizant  Deputy  Commander/assistant  commander  and 
the  Commander,  Naval  Air  Systems  Command  (AIR-00).  Specific 
procedures  follow: 

Para  4. a.  "AIR-803  will  allocate  BCE  funds  to  RFMs  in 
accordance  with  established  procedures,  using  NAVMAT  Form 
2198/1,  Administering  Office  Allocation  Distribution 
Research,  Development,  Test  and  Evaluation,  Navy  by  Program 
Year.  RCE  funds  will  be  allocated  to  a  separate  project  unit 
under  AIR-803  cognizance." 

Para  4.b.  "To  obtain  the  release  of  RCE  funds,  the  RFM 
must  submit  justification  in  the  format  enclosure  (1)  and 
forward  it  by  memorandum  to  AIR-803.  The  request  will 
include  a  financial  status  of  the  program,  complete  justifi¬ 
cation  for  the  use  of  RCE  funds  and  an  impact  statement 

explaining  the  effect  if  RCE  funds  are  not  released." 
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Para  4.c.  "AIR-803  will  review  the  request  for  accuracy 
and  substance  and  forward  it  to  the  cognizant  Deputy 
Commander/assistant  commander  or  return  it  to  the  RPM  as 
appropriate. " 

Para  4.d.  "Upon  the  recommendation  of  the  Deputy 
Commander  or  cognizant  assistant  commander,  the  request  will 
be  forwarded  to  AIR-00." 

Para  4.e.  "AIR-00  will  approve/disapprove  the  request 
and  return  it  to  AIR-803  for  appropriate  action." 

Para  4.f.  "Disapproved  requests  will  be  returned  to  the 
RFM's  by  AIR-803.  For  approved  requests,  AIR-803  will  pre¬ 
pare  the  necessary  allocation  documents  reprogramming  the  RCE 
funds  to  a  project  unit  under  the  cognizance  of  the  RFM. 
RFM's  will  be  repsonsible  for  establishing  a  task  area  code 
designated  'RCE'  in  the  Chart  of  Accounts  and  directing  RCE 
funds  to  this  task  area.  Managers  are  required  to  use  the 
limited  indicator  (block  18  of  NAVMAT  Form  7132/7,  Project 
Directive)  in  conjunction  with  the  RCE  task  area." 

Para  4.q.  "RFM's  may  indicate  at  any  time  that  RCE  funds 
are  not  needed,  in  which  case  they  will  be  reprogrammed  to 
satisfy  other  urgent  RDT&E, N  requirements.  RCE  funds  not 
used  by  RFM's  will  be  considered  for  reprogramming  after  12 
months  from  the  beginning  of  the  fiscal  year  for  which  they 
were  appropriated.  The  reprogramming  of  RCE  funds  will 
require  prior  approval  of  AIR-00  and  must  follow  established 
reprogramming  procedures  for  the  RDT&E,N  appropriation." 

o  Naval  Sea  Systems  Command  Instruction  (NAVSEAINST) 
5000. 3A.  Acquisition  Program  Appraisal  Within  the  Naval  Sea 
Systems  Command  (25  May  1982). 

Enclosure  1:  ARB  Procedures 

Para  3.  Section  III  -  Strategy  to  Achieve  Objectives  and 
Implementation  shall  contain  the  "Risk  Management  Plan  for 
dealing  with  areas  (technical,  costs,  schedule  and 
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logistics),"  and  the  business  management  plan  of  "incentives 
to  achieve  program  thresholds  including  methods  to  control 
costs,"  and  "incentives  to  improve  reliability  and  reduce 
support  cost's." 


Para  5,b.  Presentation.  "Technical,  financial,  logis¬ 
tic,  and  administrative  issues  and  program  risks  should  be 
highlighted  and  discussed."  [Beyond  this  reference,  the  docu¬ 
ment  consists  only  of  administrative  procedures  and  reporting 
or  briefing  formats.] 


o  Naval  Electronic  Systems  Command  Instruction  (NAVELEX- 
INST)  5000. 13A.  Acquisition  Program  Review  and  Reporting  Within 
the  Naval  Electronic  Systems  Command.  (19  July  1982). 


Enclosure  (2) 

5.  "The  Oversight  Group  team  will  prepare  a  report  on 
significant  risks,  ...  and  recommended  or  initiated  actions 
and  will  provide  a  ...  copy  of  this  report  to  the  Acquisition 
Manager . " 
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APPENDIX  I 


BUDGET  POLICIES  INCORPORATING  RISK  ASSESSMENT 

This  Appendix  provides  information  on  two  widely  known  Army 
budget  policies  incorporating  risk  assessment,  TRACE  and  TRACE -P. 
The  TRACE  policies  appear  separately  because,  although  governing 
only  Army  budget  matters,  they  have  become  widely  known  in  all 
three  Services,  and  Carlucci  Initiative  Number  11  directs  that, 
"...each  service  shall  review  the  TRACE  concept  and  either 
adopt  it  or  produce  an  alternative  for  their  [sic]  use."  As 
starting  points  for  the  other  services'  own  policies,  an 
understanding  of  these  concepts  will  assist  in  formulation  and 
implementation  of  similar  policies  in  the  other  two  services. 

1.  TRACE 

In  a  1974  memorandum  [7],  Ncrman  Augustine,  the  then  Assis¬ 
tant  Secretary  of  the  Army  for  Research  and  Development,  iden¬ 
tified  the  need  to  reduce  the  order  of  magnitude  of  cost  errors 
certain  to  occur  when  R&D  budgets  are  formulated  for  only  those 
eventualities  that  are  foreseen.  The  Assistant  Secretary  desired 
the  funding  of  "certain  uncertainties"  and  proposed  the  "risk 
factor"  method.  The  memorandum  provided  the  complete  concept  of 
TRACE,  including  general  guidelines  on  administration.  In  March 
1975,  the  Army  Deputy  Chief  of  Staff  for  Research,  Development, 
and  Acquisition  (DCSRDA)  published  [6],  providing  the  administra¬ 
tive  details  of  the  TRACE  system  and  outlining  the  "risk  factor" 
methodology.  Although  the  methodology  lacks  mathematical  rigor, 
it  provides  the  Army  with  a  simple,  understandable  method  pro¬ 
viding  grossly  for  the  "known  unknowns".  Prior  to  the  DCSRDA 
directive,  various  risk  assessment  tools  existed;  however,  the 
DCSRDA  directive  initiated  establishment  of  Army  and  Military 
Service  policy  for  attempting  to  determine  the  effects  of  tech¬ 
nical  risk  on  cost,  growth.  The  directive  stated  that  uncertainty 
should  be  considered  an  essential  element  of  weapons  systems 
acquisition . 


•hpmm  up? 


Reference  [6]  does  not  restrict  analysts  to  the  "risk  factor" 
method  but  allows  the  use  of  "other  generally  accepted  methodo¬ 
logies".  Some  projects  have  gon^  into  more  detail,  applying  pro¬ 
babilistic  event  analyses  and  networks  simulations.1  One  of  the 
professional  services  contractors  who  had  assisted  Augustine  in 
development  of  his  concept  briefed  various  Congressmen  and 
Congressional  staff  members  on  the  theory  and  practice  of  TRACE. 
The  briefings,  some  of  which  took  place  in  1979.  were  general!/ 
well  received. 2  Because  Army  budget  submissions  have  included 
TRACE'S  on  selected  programs  for  a  number  of  years  including 
those  following  the  cited  briefings,  there  seems  no  reason  to 
believe  that  the  Congress  is  dissatisfied  with  the  Army's  imple¬ 
mentation  of  TRACE. 

The  essential  elements  of  TRACE  are  the  following: 

o  An  analytical  assessment  of  technical  risk  resulting  in 
a  TRACE,  intended  to  be  at  the  level  at  which  there  is  as 
great  a  liklihood  of  overrun  as  underrun. 

o  Review,  within  the  Army,  of  both  the  program  manager's 
Baseline  Cost  Estimate  (BCE)  and  of  the  TRACE  estimate, 
which  is  expected  to  be  the  larger  of  the  two. 

o  Submission  of  the  TRACE,  as  modified  within  the  Department 
of  the  Army,  to  DOD  and  Congress  as  the  Program  Objectives 
Memorandum  (POM)  or  Budget  estimate. 


1  McGinnis,  LTC,  USA,  and  Kirschbaum,  Capt . ,  USAF;  "Trace 
Risk  Assessment  and  Program  Execution";  Defense  Systems 
Management  College,  Fort  Belvoir,  Va.,  Dec.  1981. 

2  Letter  to  the  author  from  John  M.  Cockerham  and  Associates, 
Inc.  Huntsville,  Ala.,  dated  21  September  1982. 
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o  Upon  approval  of  the  Budget,  release  to  the  PM  of  only  the 
amount  of  the  BCE.  Deferral  of  the  difference  between  the 
TRACE  and  the  BCE  (known  as  the  TRACE  margin  or  TRACE 
deferral)  at  the  DA  level  (DCSRDA). 

o  Release  of  any  portion  of  the  TRACE  deferral  to  the  PM 
only  upon  "request  and  approval  of  a  justified  need". 

While  the  foregoing  points  are  made  in  the  TRACE  LOI  itself, 
reference  [4]  directs  that  TRACE  deferrals  identified  for  a  given 
program  may  not  be  reprogrammed  except  under  highly  circumscribed 
conditions . 

2 .  TRACE-P 


Believing  the  TRACE  concept  and  risk  assessment  to  be  worth¬ 
while  tools,  and  finding  that  significant  cost  growth  occurs  when 
systems  transition  from  R&D  into  production,  analysts  at  The  Army 
Procurement  Research  Office  (APRO)  determined  that  a  risk 
assessment/budgeting  method  should  be  devised  to  control  that 
problem.  The  Army  investigators  identified  for  each  program  a 
set  of  11  "uncertainty  elements"  which  are  treated  as  cost  ele¬ 
ments  and  which  are  said  to  be  normally  associated  with  produc¬ 
tion  transition.  Some  (e.g.,  facilities)  are  the  same  items  as 
are  found  in  some  aggregation  of  the  BCE.  Others  (e.g.,  design 
stability)  are  more  difficult  to  relate  to  BCE  cost  elements. 
The  methodology  [8]  recommends  assigning  three  cost  values  to 
each  element,  assigning  a  distribution  to  each,  finding  the 
distribution  mean  of  each,  and  subtracting  the  relevant  BCE  ele¬ 
ment  costs  from  the  mean  costs  to  determine  the  TRACE-P  deferral. 
Note  that  in  contrast  to  the  policy  with  TRACE  for  R&D,  tnis 
reference  recommends  use  of  the  mean  instead  of  the  median 
(identical  to  the  50  percent  confidence  level),  however  it  also 
states  that  confidence  levels  for  TRACE-P  estimates  can  be 
obtained  if  other  supplementary  techniques  are  used. 

The  Army  DCSRDA  has  not  yet  approved  a  TRACE-P  procedure, 
pending  further  evaluation;  however,  the  Development  and  Readi- 
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ness  Command  Comptroller  has  signed  a  DARCOM  Letter  of 
Instruction  15]  which  defines  a  modified  methodology  (compared  to 

that  in  [8])  and  the  same  administrative  procedures  for  TRACE-P 
as  for  TRACE  for  R&D . 
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APPENDIX  3 


CENTERS  OF  RESEARCH 


The  following  is  a  list  of  some  of  the  more  prominent  organi¬ 
zations  engaged  in  research  on,  and  accomplishment  of,  acquisi¬ 
tion  program  risk  assessment. 

1 .  Department  of  Defense 

o  Defense  Systems  Management  College,  Fort  Belvoir,  VA. 


2 .  U.S.  Army 

o  U.S.  Army  Logistics  Management  Center,  Fort  Lee,  VA 

o  Army  Procurement  Research  Office,  U.S.  Army  Systems 
Analysis  Activity,  Fort  Lee,  VA 

o  DARCOM  Material  Systems  Analysis  Activity,  Aberdeen 
Proving  Ground,  MD. 

o  U.S.  Army  Armament  Command,  Rock  Island  Arsenal,  Rock 
Island,  IL. 

o  U.S.  Army  Construction  Engineering  Research 
Laboratory,  Champaign,  IL. 

o  U.S.  Army  Aviation  Research  and  Development  Command, 
St.  Louis,  MO. 

o  U.S.  Army  Material  Readiness  Command,  Systems  Analysis 
Directorate,  Rock  Island,  IL. 

o  U.S.  Army  Research  Office,  Research  Triangle  Park,  N.C 

o  U.S.  Army  Missile  Command,  Redstone  Arsenal,  AL. 

o  U.S.  Army  Electronics  Command,  Systems  Analysis 
Office,  Fort  Monmouth,  N.J. 

o  U.S.  Army  War  College,  Carlisle  Barracks,  PA. 

o  Red  River  Army  Depot,  Texarkana,  TX. 

o  U.S.  Army  Aviation  Systems  Command,  St.  Louis,  MO. 


3.  U.S.  AIR  FORCE 


o  Air  Command  and  Staff  College,  Maxwell  AFB,  AL. 

o  School  of  Engineering,  Air  Force  Institute  of 
Technology,  Wr ight-Patterson  AFB,  OH. 

o  The  School  of  Systems  and  Logistics,  Air  Force 

Institute  of  Technology,  Wright-Patterson ,  AFB,  OH. 

o  Air  Force  Business  Management  Research  Center, 
Wright-Patterson  AFB,  OH. 

o  Air  War  College,  Maxwell  AFB,  AL. 

4.  U.S.  NAVY 

o  Navy  Office  for  Acquisition  Research,  Naval  Material 
Command  Headquarters,  (MAT-08P) ,  Washington,  DC. 

5.  CIVILIAN  -  ACADEMIC 

o  Carnegie-Mellon  University,  Pittsburg,  PA. 

o  Northwestern  University,  Evanston,  IL. 

o  The  Pennsylvania  State  University,  University  Park, 
PA. 

o  Operations  Research  Center,  University  of  California, 
Berkeley,  CA. 

o  University  of  Oklahoma,  Norman,  OK. 

o  University  of  Southern  California,  Colorado  Springs, 
CO. 

o  Clemson  University,  Department  of  Mathematical 
Sciences,  Clemson,  SC. 

6 .  CIVILIAN  -  CORPORATE 

o  Hughes  Aircraft  Company 
o  Decisions  and  Design,  Inc.,  McLean,  VA. 
o  General  Research  Corporation,  Santa  Barbara,  CA. 
o  Decision  Research,  Eugene,  OR. 
o  Martin-Marietta  Corp.,  Orlando,  FL. 
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o  Mathematica,  Pruic  ton,  NJ. 

o  System  Development  Corporal,  ion ,  McLean,  VA. 

o  Meridian  Corp.,  Falls  Church,  VA. 

o  The  Rand  Corporation,  Santa  Monica,  CA. 

o  Grumman  Aerospace  Corporation  Operations  Analysis, 

New  York 

o  LOG/AN  Inc.,  Los  Angeles,  CA. 
o  JMCA  &  Associates,  Huntsville,  AL. 

o  Management  Consulting  and  Research,  Inc.,  McLean,  Va . 
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APPENDIX  K 


APPLICATION  TO  CONTRACT  STRUCTURE 


Any  method  providing  cost  risk  in  terms  of  a  cumulative 
distribution  function  can  be  used  in  contract  share  ratio  deter¬ 
mination  as  discussed  in  [1],  The  reference  employs  a  contract 
cost  model  including  target  cost,  target  price,  and  profit.  The 
program  manager  selects  a  point  above  which  the  contractor  should 
assume  full  risk  and  less  than  which  the  government  and  contrac¬ 
tor  share.  He  selects  the  point  based  on  his  decision  that  the 
contractor  should  assume  full  risk  above  a  cost  representing  a 
confidence  level  he  determines.  Figure  [K-l]  illustrates  such  an 
application.  For  the  example  in  Figure  [K-l],  cumulative  proba¬ 
bility  is  determined  indirectly  b^  assuming  the  overall  cost  to 
be  representable  by  a  normal  distribution,  so  that  the  mean  plus 
a  number  of  standard  deviations  determines  the  cumulative  proba¬ 
bility.  If  the  normal  distribution  is  considered  too  gross  an 
approximation,  any  distribution  with  skew  could  be  used,  and  the 
policy  decision  can  be  based  on  the  50£Ji  and  a  higher  selected 
confidence  level,  for  example  the  90iJl.  In  other  words,  the 
contractor  would  assume  full  risk  above  the  higher  level,  and  the 
government  and  contractor  share  cost  between  that  and  the  point 
at  which  there  is  an  equal  likelihood  of  cost  being  overrun  or 
underrun.  The  ratio  itself  is  determined  by  using  the  Defense 
Acquisition  Regulations  Weighted  Guidelines  to  fix  the  profit  at 
the  percentile  point. 
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Figure  K-l 

INCENTIVE  CONTRACT 

The  symbols  in  the  figure  are 

defined : 

TP  is  Target  Profit 

WP  is  Warranted  Profit 

TC  is  Target  Cost 

RAC  is  Risk  Analysis  Cost 

O  .  r 


T 


Here  E(TC)  is  Expected  (mean)  Target  Cost  and  3  V VAR  ( TCf) 
is  three  times  the  standard  deviation  of  Target  Cost.  A 
variation  of  this  diagram  can  be  used  to  investigate  the  cost  to 
the  government  of  share  ratio  changes. 


While  Figure  K-l  is  not  model  output  in  the  sense  of  a  com¬ 
puter  printout,  it  exemplifies  an  application  of  risk  assessment 
output  information. 
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APPENDIX  L 


APPLICATIONS  OP  METHODS  OUTPUTS 


Chapters  II  and  III  discussed  the  types  of  outputs  produced 
by  risk  assessments  and  how  they  could  be  used  in  program  manage¬ 
ment  decisions.  Chapter  IV  summarized  the  assessment  methods  and 
briefly  presented  the  output  of  each.  Table  L-l  summarizes  the 
connection  of  method  to  application  via  method  output.  By 
reading  across  from  a  specific  method  at  the  left,  the  reader 
finds  an  entry  under  the  category  of  information  which  the  method 
supplies  from  routine  output,  from  improved  systems,  or  in 
limited  fashion.  The  reader  then  follows  down  the  column  to  the 
intersection  with  the  row  extending  from  a  selected  application. 
An  X  entry  signals  that  such  information  is  useful  in  decisions 
relating  to  that  application . ^ 

In  considering  Table  L-l,  it  helps  to  realize  chat  some  of 
the  columns  identify  types  of  information  inherent  in  others.  For 
example,  a  probability  density  function  inherently  includes  an 
intuitive  understanding  of  skew  and  the  coefficient  of  variation, 
respectively.  The  CDF  inherently  includes  confidence  level 
information . 


It  can  be  seen  from  Table  L-l  that  the  mean  as  a  category  of 
information  is  not  used  in  decision  categories  by  the  methods 
discussed  herein.  This  is  because  most  output  distributions  will 
be  skewed  and  the  mean  does  not  specify  a  confidence  level  for 
such  distributions.  For  the  mean  to  be  of  value  requires  more 
intensive  understanding  of  the  decision  analysis  [41,  151.  The 
information  wanted,  however,  lies  in  the  other  output  categories. 


^Assignment  of  output  to  application  is  admittedly  subjective  and 
subject  to  change  as  researchers  explore  new  applications. 
Managers  must  evaluate  these  assignments  by  their  own  criteria. 


In  Table  L-l  entries  under  the  heading  INTERNAL  indicated 
that  risk  information  is  provided  on  program  elements.  Entries 
under  COMPREHENSIVE  indicate  that  such  information  is  provided  on 
the  program  as  a  whole. 

The  joint  cumulative  distribution  function  shows  the  cumula¬ 
tive  probability  of  any  pair  of  two  variables,  thus  permitting 
the  manager  to  perform  a  trade-off  between  the  two.  For  example, 
if  cost  and  schedule  have  a  joint  cumulative  distribution, 
the  manager  can  determine  the  probability  that  cost  will  be 
less  than  or  equal  to  X  and  development  time  less  than  or  equal 
to  Y .  If  he  prefers  a  lower  cost,  he  can  see  that  there  is  a 
lower  probability  of  cost  less  than  or  equal  to  (X-a)  and  deve¬ 
lopment  time  less  than  or  equal  to  Y.  Alternatively,  he  can  find 
a  point  (X-a)  and  (Y+b)  such  that  the  probability  of  cost  less 
than  or  equal  to  (X-a)  and  development  time  less  than  or  equal 
to  (Y+b)  is  the  same  as  for  his  original  pair,  X  and  Y. 
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